Markdown

CUPTI: the profiling interface under Nsight, PyTorch and DCGM¶

Scope: CUPTI as the layer every GPU profiler is built on, and therefore as a thing you administer rather than call. Who is allowed to read the GPU's performance counters (NVreg_RestrictProfilingToAdminUsers, and the capabilities system replacing it), why only one tool can hold those counters at a time (which is why dcgm-exporter and an engineer's nsys run fight), what tracing costs, and how to turn an activity buffer into a number that is not a lie. For the tools built on it, see Nsight profiling workflow; for the management library that is not part of this stack, see NVML.

What it is¶

"The CUDA Profiling Tools Interface (CUPTI) provides C and Python APIs for building profiling and tracing tools for CUDA applications, providing detailed insight into how code executes on both the CPU and GPU."¹ It ships with the CUDA Toolkit, not the driver: "The CUPTI SDK is part of the CUDA Toolkit, and will be installed along all the other CUDA libraries", under /usr/local/cuda-<version>/extras/CUPTI/, and it is not added to LD_LIBRARY_PATH for you.¹

The docs draw a line that governs everything downstream, and it is worth internalizing before touching permissions or overhead:

"tracing means collecting timestamps and associated metadata for CUDA activities such as runtime and driver API calls, kernel launches, and memory copy operations during application execution."

"profiling refers to collecting GPU performance metrics for individual kernels or sets of kernels, often in isolation from the rest of the workload. Profiling may require replaying kernels or even the full application multiple times to gather all desired metrics under controlled conditions."¹

Tracing is cheap, unprivileged, and mostly composable. Profiling is expensive, privileged, and exclusive. Nearly every operational surprise in this page follows from that one distinction.

The current API set:¹

API	What it does
Activity	"Asynchronously record CUDA activities, e.g. CUDA API, Kernel, memory copy"
Callback	"CUDA event callback mechanism to notify subscriber that a specific CUDA event executed e.g. 'Entering CUDA runtime memory copy'"
Host Profiling	"Host APIs for enumeration, configuration and evaluation of performance metrics"
Range Profiling	"Target APIs for collection of performance metrics for a range of execution"
PC Sampling	"Sampling of the warp program counter and warp scheduler state (stall reasons)"
SASS Metrics	"Collect kernel performance metrics at the source level using SASS patching"
PM Sampling	"Collect hardware metrics by sampling the GPU performance monitors (PM) periodically at fixed intervals"
Checkpoint	"Provides support for automatically saving and restoring the functional state of the CUDA device"
Profiling	Deprecated in CUDA 13.0; use Range Profiling

The old Event API and Metric API are gone, not deprecated: in CUDA 13.0 "The CUPTI Event API from the header cupti_events.h and the CUPTI Metric API from the header cupti_metrics.h are dropped. Calling any Event or Metric API will return the error code CUPTI_ERROR_LEGACY_PROFILER_NOT_SUPPORTED."³ Any tool still on them stops working on CUDA 13, and the failure is a clean error code rather than a crash, which is at least a courtesy.

Who is built on it. Nsight Systems: "Nsight Systems uses CUPTI for CUDA profiling, including to collect the CUDA memory usage by the application processes."¹⁰ The frameworks: CUPTI's own troubleshooting text names "a framework with CUPTI integration (like PyTorch or TensorFlow)" as a CUPTI client,¹ and PyTorch's kineto states "NVIDIA CUPTI: used to collect traces and metrics from NVIDIA GPUs."¹¹ HPCToolkit wraps it too, and it is the standard integration point for third-party HPC profilers. Nsight Compute is the interesting exception: no NVIDIA source says it is CUPTI-based, and its docs treat "a client of CUPTI's Profiling API" as a separate process competing for the same reservation, so do not assume it.⁹

Why it matters to someone who never calls it¶

You will most likely never write a cuptiSubscribe. You will still administer CUPTI, because every torch.profiler call, every nsys run, and every DCGM profiling metric on your cluster goes through its permission model and its exclusivity rules. Two consequences dominate the ticket queue:

Profiling is admin-only by default, and the error message does not say why. A user runs nsys profile or ncu and gets:

"ERR_NVGPUCTRPERM: The user running \<tool_name/application_name> does not have permission to access NVIDIA GPU Performance Counters or the Hardware Event System on the target device."⁶

This is not a bug, it is a security mitigation, applied since "driver versions 419.17+ on Windows or 418.43+ on Linux."⁶

Only one client can hold the counters. DCGM says it plainly: "Due to current hardware limitations, collection of profiling metrics with DCGM will conflict with usage of other developer tools from NVIDIA such as Nsight Systems or Nsight Compute."⁸ If you run dcgm-exporter with profiling metrics enabled on your nodes, you have already taken the lock. Be careful about which error you go looking for, because the two sides report differently: the engineer's tool says the reservation failed (Nsight Compute: "Profiling failed because a driver resource was unavailable", and a CUPTI client gets CUPTI_ERROR_HARDWARE_BUSY, "The performance monitoring hardware is currently in use by another client"), while the Error setting watches ... the affected resource is in use string is what DCGM prints when a developer tool got there first.⁹¹⁸

When to use it (and when not)¶

Goal	Reach for	Not
Find the bottleneck in a training step	`nsys` / `torch.profiler`	Writing a CUPTI tool; they are already CUPTI tools
Continuous, low-cost timeline data in your own harness	CUPTI Activity API	Range Profiling; counters are not free
Per-kernel counters (SM active, tensor active, memory throughput)	Range Profiling API (or `ncu`)	The Event/Metric APIs; dropped in CUDA 13.0
Fleet-wide utilization metrics	DCGM profiling metrics	Per-node CUPTI tools; DCGM already holds the counters
Device state, ECC, Xid, power	NVML	CUPTI; it does not do device management
Correctness bugs	Compute Sanitizer	CUPTI, which cannot even run at the same time

Do not collect counters on a shared production GPU. Counter collection may serialize kernels and replay them, and it takes an exclusive reservation that blocks your own monitoring. Tracing is the mode that is safe to leave running; profiling is a scheduled, isolated activity.

Three hard incompatibilities to design around:

Not with the debugger or the sanitizer. "CUPTI still does not support tracing or profiling along with other NVIDIA Developer Tools like cuda-gdb or Compute Sanitizer."¹ This is the same wall from the other side of cuda-gdb's rule that core dumps are unsupported when another tool is attached.
Not on a shared MIG compute instance. For a shared compute instance, "Due to this resource sharing, collecting profiling data from shared units is not permitted. Attempts to collect metrics from a shared unit will result in NaN values." Tracing still works, and profiling works on an isolated compute instance.⁵
Not under MPS, at least for the (now deprecated) Profiling API, which is "not supported under MPS (Multi-Process Service), Confidential Compute, or SLI configured systems."¹ Note the scope honestly: that sentence sits in the Profiling API section, and NVIDIA states no equivalent MPS restriction for the Range Profiling API that replaces it, so do not conclude counter collection under MPS is categorically impossible. Verify on your driver.

Architecture¶

flowchart TB
    APP["CUDA application<br/>(PyTorch, vLLM, your kernel)"] --> CUPTI["CUPTI (CUDA Toolkit)"]
    CUPTI --> TRACE["TRACE path<br/>Activity + Callback APIs<br/>cheap, unprivileged, composable"]
    CUPTI --> PROF["PROFILE path<br/>Range Profiling / PM Sampling<br/>replay, serialize, exclusive"]
    TRACE --> BUF["activity buffers<br/>(unordered, correlationId-linked)"]
    PROF --> PERM{"NVreg_RestrictProfilingToAdminUsers<br/>or nvidia-capabilities (R610+)"}
    PERM -->|"denied"| ERR["ERR_NVGPUCTRPERM"]
    PERM -->|"allowed"| PM["GPU performance monitors"]
    DCGM["DCGM (dcgm-exporter)"] --> PM
    NCU["Nsight Compute"] --> PM
    PM --> LOCK["one reservation holder at a time"]

The bottom of the diagram is the whole operational problem: dcgm-exporter, Nsight Compute, and any CUPTI Profiling client all want the same single-holder resource. The trace path, above it, does not.

How to administer it: the permission model¶

This is the part that belongs to the platform team, not the user.

The classic control is a kernel module parameter. To open profiling to all users:⁶

# Legacy regkey method. Create /etc/modprobe.d/nvidia-profiling.conf:
options nvidia NVreg_RestrictProfilingToAdminUsers=0

A reboot "may be required" for the change to take effect, and on some systems the initramfs must be rebuilt or the file is ignored: dracut --regenerate-all -f on Red Hat derivatives, update-initramfs -u -k all on Debian derivatives.⁶ Verify what the running module actually has, rather than what your .conf says:

grep RmProfilingAdminOnly /proc/driver/nvidia/params
# 1 = admins only, 0 = all users

One caveat on newer drivers, straight from the same page: "On R610+ drivers, this flag only reflects the regkey state and does not account for capability-based access grants."⁶ So a 1 there does not prove your users cannot profile; they may hold a capability instead.

What is the default? NVIDIA's own header contradicts itself, so trust the code. In open-gpu-kernel-modules at driver 610.43.03, the comment above the parameter says "0: Do not restrict GPU counters (default)", while the actual definition, five hundred lines further down the same file, is NV_DEFINE_REG_ENTRY(__NV_RM_PROFILING_ADMIN_ONLY_PARAMETER, 1);, and that macro expands to a variable initialized to its second argument.⁷ The compiled-in default is 1, restricted, which is also what the ERR_NVGPUCTRPERM page implies when it says a recent driver installation "has disabled access to GPU Performance Counters for regular users". The comment is stale. If you grep the header and conclude profiling is open by default, you have been misled by NVIDIA's documentation, not by your cluster.

The alternatives to opening it globally:

Capabilities. A user with CAP_SYS_ADMIN may profile; "Starting in driver version R565, the CAP_PERFMON capability will also allow access."⁶ CAP_PERFMON is the narrower grant and generally the better one, with one documented exception: "CAP_PERFMON will not work in secure execution mode unless profiling within a container as described above."⁶
Containers. "When profiling within a container, access must be enabled on the host, or the container must be started with the appropriate permissions by passing --cap-add=SYS_ADMIN as an admin user."⁶
The R610+ replacement. Newer drivers move to per-capability device nodes: profiler-context ("Grants profiling access to intra-context scope"), profiler-device (adds device-level scope), and trace-device. NVIDIA states the regkey method is legacy and that "In a future release, the regkey-based method will be removed."⁶ Note the scoping limit: "Profiling capabilities are granted system-wide and cannot be scoped to individual GPUs."⁶ You cannot grant profiling on GPU 3 only.

The right posture for a multi-tenant cluster is usually: leave counters restricted on production nodes, and grant profiling (via CAP_PERFMON or a capability-enabled debug pod) on a dedicated pool where the serialization and exclusivity cost nobody else anything.

How to develop with it: reading an activity buffer honestly¶

The Activity API is asynchronous and buffer-based. You register two callbacks with cuptiActivityRegisterCallbacks, one that hands CUPTI an empty buffer and one that receives a full one, and "For typical workloads, it's suggested to choose a size between 1 and 10 MB."¹ Two properties of what arrives will break a naive tool:

The buffer is unordered. "CUPTI doesn't guarantee any ordering of the activities in the activity buffer as activity records for few activity kinds are added lazily."¹ Any logic that assumes records arrive in time order is wrong.
The link back to the launch site is a correlation ID, and it is opt-in. "Every CUDA driver/runtime API invocation that CUPTI records is assigned a unique correlation ID", the resulting kernel and memcpy records "carry the same correlation ID", and critically: "For CUPTI to generate correlationId values, the CUDA API activity kinds i.e. CUPTI_ACTIVITY_KIND_RUNTIME and/or CUPTI_ACTIVITY_KIND_DRIVER must be enabled."¹ Forget that, and you get kernel records that cannot be attributed to any line of code.

Now the arithmetic. The question every trace consumer asks first is "how busy was the GPU", and the obvious implementation, summing kernel durations, is wrong the moment two streams overlap. It can report more than 100% busy, which should be the tell. The correct quantity is the union of the kernel intervals, and it must be computed without assuming buffer order.

"""Turning a CUPTI activity buffer into GPU busy time and launch-gap analysis.

Two documented properties of the buffer drive every assertion here:
  1. "CUPTI doesn't guarantee any ordering of the activities in the activity buffer"
  2. a kernel record is tied to its launching API call only by correlationId, and only
     if CUPTI_ACTIVITY_KIND_RUNTIME / _DRIVER were enabled.
Timestamps are in nanoseconds.
"""
import numpy as np

# (correlationId, stream, start_ns, end_ns) for CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL
KERNELS = np.array([
    (11, 7, 1_000, 5_000),     # gemm      on stream 7
    (12, 7, 5_200, 9_000),     # gemm      on stream 7, after a 200 ns launch gap
    (13, 13, 3_000, 4_000),    # allreduce on stream 13, OVERLAPS the first gemm
    (14, 13, 4_000, 6_000),    # allreduce on stream 13, touches 13's end, overlaps 12
    (15, 7, 9_000, 9_000),     # a kernel that recorded a zero-length duration
], dtype=np.int64)

# (correlationId, start_ns, end_ns) for CUPTI_ACTIVITY_KIND_RUNTIME (the launch calls)
RUNTIME = np.array([
    (11, 900, 950),
    (12, 5_050, 5_100),
    (13, 2_900, 2_950),
    (14, 3_950, 3_990),
    (15, 8_900, 8_950),
], dtype=np.int64)

def union_busy(iv):
    """Total wall time during which at least one kernel was resident. Order-independent."""
    if len(iv) == 0:
        return 0
    iv = iv[np.argsort(iv[:, 0], kind="stable")]          # sort by start; the buffer is unordered
    total, cur_s, cur_e = 0, iv[0, 0], iv[0, 1]
    for s, e in iv[1:]:
        if s > cur_e:                                     # a real gap: close the run
            total += cur_e - cur_s
            cur_s, cur_e = s, e
        else:
            cur_e = max(cur_e, e)                         # overlap or touch: extend
    return int(total + cur_e - cur_s)

def naive_busy(iv):
    """What a per-kernel sum reports. Double counts every overlap."""
    return int((iv[:, 1] - iv[:, 0]).sum())

iv = KERNELS[:, 2:4]
span = int(iv[:, 1].max() - iv[:, 0].min())               # wall clock across the trace

# --- Happy path: one stream, no overlap, sum and union agree ---------------
serial = KERNELS[KERNELS[:, 1] == 7][:, 2:4]
serial = serial[:2]                                       # the two gemms, disjoint
assert union_busy(serial) == naive_busy(serial) == 7_800
print(f"single stream, disjoint kernels: union == sum == {union_busy(serial)} ns")

# --- Adversarial 1: two streams overlap; the naive sum exceeds wall clock --
u, n = union_busy(iv), naive_busy(iv)
assert u == 8_000, u                                      # 1000..9000 is continuously busy
assert n == 10_800, n                                     # 4000 + 3800 + 1000 + 2000 + 0
assert n > span == 8_000                                  # a "GPU busy" of 135% of the window
assert u <= span                                          # the union never can
print(f"2 streams: union={u} ns, naive sum={n} ns, wall span={span} ns "
      f"-> naive reports {100 * n / span:.0f}% busy, which is impossible")

# --- Adversarial 2: the buffer is unordered, so the result must not depend on order
rng = np.random.default_rng(7)
for _ in range(200):
    shuffled = iv[rng.permutation(len(iv))]
    assert union_busy(shuffled) == u
print(f"200 random buffer orderings: union invariant at {u} ns")

# --- Adversarial 3: touching intervals are not a gap, and zero-length is not busy
touching = np.array([[0, 100], [100, 200]], dtype=np.int64)
assert union_busy(touching) == 200                        # 3000-4000 and 4000-6000 above
zero = np.array([[9_000, 9_000]], dtype=np.int64)
assert union_busy(zero) == 0                              # kernel 15 adds nothing
assert union_busy(np.vstack([iv, zero])) == u             # and cannot inflate the total
print("touching intervals merge (no phantom gap); a zero-length kernel adds 0 ns")

# --- Adversarial 4: idle gaps are the launch overhead you came to find -----
merged, order = [], iv[np.argsort(iv[:, 0], kind="stable")]
cs, ce = order[0]
for s, e in order[1:]:
    if s > ce:
        merged.append((cs, ce)); cs, ce = s, e
    else:
        ce = max(ce, e)
merged.append((cs, ce))
gaps = [int(merged[i + 1][0] - merged[i][1]) for i in range(len(merged) - 1)]
assert merged == [(1_000, 9_000)], merged                 # one continuous busy run
assert gaps == []                                         # nothing idle inside the window
assert span - u == 0
# Now drop the allreduce stream: the gemm-only timeline exposes the 200 ns launch gap.
gemm = KERNELS[(KERNELS[:, 1] == 7) & (KERNELS[:, 3] > KERNELS[:, 2])][:, 2:4]
gemm_span = int(gemm[:, 1].max() - gemm[:, 0].min())
gemm_idle = gemm_span - union_busy(gemm)
assert gemm_idle == 200, gemm_idle                        # 5000 -> 5200
print(f"stream 7 alone: {gemm_idle} ns idle inside an {gemm_span} ns window "
      f"({100 * gemm_idle / gemm_span:.1f}% of it), the launch gap the union exposes")

# --- Adversarial 5: correlationId is the only link back to the launch site -
def attribute(kernels, api_records):
    """Join kernel records to their launching API call. Must tolerate missing records."""
    by_corr = {int(c): (int(s), int(e)) for c, s, e in api_records}
    launched, orphans = {}, []
    for corr, _, k_start, _ in kernels:
        rec = by_corr.get(int(corr))                      # .get, not [], is the whole point
        if rec is None:
            orphans.append(int(corr))
            continue
        api_end = rec[1]
        assert api_end <= k_start                         # the launch precedes the execution
        launched[int(corr)] = int(k_start) - api_end      # launch-to-execute queue latency
    return launched, orphans

latency, orphans = attribute(KERNELS, RUNTIME)
assert latency == {11: 50, 12: 100, 13: 50, 14: 10, 15: 50} and orphans == []

# A buffer can be delivered with only some API records present: the client may have
# enabled only CUPTI_ACTIVITY_KIND_DRIVER, or dropped records by returning buffers late.
partial = RUNTIME[[0, 2, 4]]                              # correlations 11, 13, 15 survive
latency_p, orphans_p = attribute(KERNELS, partial)
assert sorted(orphans_p) == [12, 14]                      # these kernels have no launch site
assert set(latency_p) == {11, 13, 15}
missing = None
try:                                                      # the naive join a tool actually ships
    {int(c): dict((int(a), (int(s), int(e))) for a, s, e in partial)[int(c)] for c in KERNELS[:, 0]}
    raise AssertionError("expected a KeyError from the unguarded join")
except KeyError as exc:
    missing = int(str(exc))
assert missing == 12
print(f"correlationId join: {len(latency)} kernels attributed (queue latencies "
      f"{sorted(latency.values())} ns); with API records partly missing, {len(orphans_p)} "
      f"kernels orphan and an unguarded dict lookup raises KeyError({missing})")

Executed output:

single stream, disjoint kernels: union == sum == 7800 ns
2 streams: union=8000 ns, naive sum=10800 ns, wall span=8000 ns -> naive reports 135% busy, which is impossible
200 random buffer orderings: union invariant at 8000 ns
touching intervals merge (no phantom gap); a zero-length kernel adds 0 ns
stream 7 alone: 200 ns idle inside an 8000 ns window (2.5% of it), the launch gap the union exposes
correlationId join: 5 kernels attributed (queue latencies [10, 50, 50, 50, 100] ns); with API records partly missing, 2 kernels orphan and an unguarded dict lookup raises KeyError(12)

The 135% is the point. A tool that sums kernel durations across streams reports a GPU busier than wall clock, which is impossible, and nobody notices because the number merely looks high rather than absurd. The same union, inverted, is the useful measurement: the gaps between merged intervals are the launch bubbles, and on stream 7 alone they are 2.5% of the window. That is the quantity comms-compute overlap work is trying to drive to zero.

How to maintain it: versions and the driver pairing¶

CUPTI is versioned with the CUDA Toolkit, and it is one of the few components you may need to update between toolkit releases: "Normally packaged with the CUDA Toolkit, NVIDIA occasionally uses this page to provide CUPTI improvements and bug fixes between toolkit releases."² cuptiGetVersion reports the loaded version at runtime.

The compatibility direction that is documented is old tool, new driver: CUPTI "adheres to CUDA Toolkit compatibility requirements with CUDA drivers, which includes support for Backward, Forward and Enhanced compatibilities", and "A profiling tool based on an older version of CUPTI can still operate with a more recent CUDA driver."¹ The reverse direction, a newer CUPTI against an older driver, is not promised in the same sentence; the docs send you to the toolkit's minimum-driver table, and "Attempting to use CUPTI calls with an incompatible CUDA driver version will result in a CUPTI_ERROR_NOT_INITIALIZED error code."¹ The CUDA 13.3 multiple-subscriber feature is a concrete instance: a new CUPTI capability that requires driver r610+ regardless of toolkit.

Maintenance work that is worth scheduling rather than discovering during an incident:

Audit for the dropped APIs before a CUDA 13 rollout. Anything still calling the Event or Metric API returns CUPTI_ERROR_LEGACY_PROFILER_NOT_SUPPORTED and collects nothing.³ This includes vendored or unmaintained internal profilers, which is where it usually hides.
Pin the driver alongside the toolkit the same way you pin everything else on the node (driver install and lifecycle). A CUPTI feature that silently no-ops on an older driver is worse than one that fails loudly.
Re-check the permission mechanism when you move to R610 or later. The regkey still works, but NVIDIA has said it goes away, and the capabilities system that replaces it is configured differently.⁶

How to run it in production¶

Pause your fleet monitoring before anyone profiles. DCGM's own guidance, with the error you will see first:⁸

# The symptom, from dcgmi or the API:
#   Error setting watches. Result: The requested operation could not be completed
#   because the affected resource is in use.

dcgmi profile --pause      # release the counters
# ... run nsys / ncu / your CUPTI tool ...
dcgmi profile --resume

The programmatic equivalents are dcgmProfPause() and dcgmProfResume(). Note the consequence for your dashboards: "When paused, DCGM will publish BLANK values for profiling metrics", testable with DCGM_FP64_IS_BLANK(value).⁸ A profiling session therefore punches a hole in your utilization graphs, and an alert rule that treats a missing value as zero will fire. Nsight Compute suggests exactly this workflow from its side: "If you expect the problem to be caused by DCGM, consider using dcgmi profile --pause to stop its monitoring while profiling with NVIDIA Nsight Compute."⁹

Understand what serializes. Not all tracing is equal:

CUPTI_ACTIVITY_KIND_KERNEL (serial kernel trace) "can significantly change the overall performance characteristics of the application because all kernel executions are serialized on the GPU."⁴
CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL "doesn't affect the concurrency of the kernels in the application", but CUPTI instruments the kernel code to do it, so it costs more per launch.⁴

For a serving or training workload, CONCURRENT_KERNEL is the correct choice: serializing kernels does not just slow the app, it changes the thing you are trying to measure. Nsight Compute, separately, "serializes kernel launches within the profiled application, potentially across multiple processes profiled by one or more instances of the tool at the same time", because "some GPU and driver objects can only be acquired by a single process at a time... meaning only one process can profile a given device at a time."⁹

Expect no single headline overhead number; expect per-feature ones. CUPTI's own statement is qualitative: overhead "largely depends on the density of the CUDA activities in the application", and "In general overhead of tracing i.e. activity APIs is much lesser than the metrics profiling APIs."⁴ Individual features do carry published figures, though: Nsight Systems warns that "Collecting Unified Memory CPU page faults can cause overhead of up to 70% in testing. Use this functionality only when needed."¹⁰ And HPCToolkit, measuring CUPTI directly, reports two: "we have seen the execution time double when profiling and tracing a GPU-accelerated application that launches kernels very frequently", and separately that for PC sampling "the overhead is less than 5x".¹² So the honest budget is 2x for launch-heavy tracing, and up to an order of magnitude once you turn on sampling or counters. Measure your own; do not extrapolate one figure across features.

Multiple tracers: check your CUDA version before you promise it. This changed recently and the documentation has not fully caught up. As of CUDA 13.3, "CUPTI supports multiple concurrent subscribers to trace CUDA activities simultaneously", requiring "CUDA 13.3+ and driver r610+".¹ Before that, a second subscriber got CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED: with Nsight Systems and PyTorch both documented as CUPTI clients, that is the mechanism behind the familiar failure of nesting one inside the other.¹ The limits still bite: it "applies only to the CUPTI Activity API. It is not applicable to CUPTI Profiling APIs", and V1 and V2 API styles cannot be mixed in one process.¹ Be aware that CUPTI's own troubleshooting section still describes the single-subscriber rule as current, contradicting its multiple-subscribers section in the same document; the multiple-subscribers section is the newer text.¹

Failure modes¶

Symptom	Cause	Fix
`ERR_NVGPUCTRPERM` on `nsys` / `ncu` / `torch.profiler` counters	Counter access is restricted to admins by default since driver 418.43⁶	`NVreg_RestrictProfilingToAdminUsers=0` plus initramfs rebuild, or grant `CAP_PERFMON`, or use the R610+ capabilities
The `.conf` file is right but nothing changed	The initramfs still carries the old parameters, or no reboot⁶	`dracut --regenerate-all -f` or `update-initramfs -u -k all`, then confirm `RmProfilingAdminOnly` in `/proc/driver/nvidia/params`
You read the driver header and concluded profiling is open by default	The comment says default 0; the code compiles in 1⁷	Trust `/proc/driver/nvidia/params`, not the comment
Your profiler reports "a driver resource was unavailable", or CUPTI returns `CUPTI_ERROR_HARDWARE_BUSY`	Something else holds the counter reservation, commonly `dcgm-exporter`⁹¹	`dcgmi profile --pause`, profile, `dcgmi profile --resume`
DCGM logs `Error setting watches ... the affected resource is in use`	The reverse case: a developer tool holds the counters and DCGM cannot watch them⁸	Finish the profiling session; DCGM resumes on its own watches
Utilization panels go blank during a profiling session	DCGM publishes `BLANK` values while paused⁸	Expect it; do not let an alert rule read blank as zero
Metrics come back as `NaN` under MIG	The compute instance is shared, and profiling shared units is not permitted⁵	Profile on an isolated compute instance; tracing works either way
Profiler and `cuda-gdb` or Compute Sanitizer refuse to coexist	"CUPTI still does not support tracing or profiling along with other NVIDIA Developer Tools like cuda-gdb or Compute Sanitizer."¹	Run one tool at a time
A tool built on the Event/Metric API dies on CUDA 13	Both APIs were dropped; calls return `CUPTI_ERROR_LEGACY_PROFILER_NOT_SUPPORTED`³	Port to the Range Profiling API
The app got much slower under tracing and the timeline looks serial	`CUPTI_ACTIVITY_KIND_KERNEL` serializes all kernel execution⁴	Use `CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL`
"GPU busy" exceeds 100%	Kernel durations were summed across overlapping streams (executed model above)	Take the union of intervals, not the sum
Kernel records cannot be traced back to any code	`CUPTI_ACTIVITY_KIND_RUNTIME` / `_DRIVER` were not enabled, so no correlation IDs¹	Enable the CUDA API activity kinds

References¶

NVIDIA, CUPTI documentation (v2026.2.1, last updated 2026-06-12): https://docs.nvidia.com/cupti/
NVIDIA, CUPTI product page (packaging, out-of-band updates, feature list): https://developer.nvidia.com/cupti
NVIDIA, CUPTI Usage (tracing vs profiling, activity buffers, correlation IDs, multiple subscribers, overhead): https://docs.nvidia.com/cupti/main/main.html
NVIDIA, CUPTI release notes (Event/Metric API dropped in CUDA 13.0): https://docs.nvidia.com/cupti/release-notes/release-notes.html
NVIDIA, CUPTI special configurations (MIG, vGPU, WSL): https://docs.nvidia.com/cupti/special-configurations/special-configurations.html
NVIDIA, ERR_NVGPUCTRPERM: permission issue with performance counters: https://developer.nvidia.com/nvidia-development-tools-solutions-ERR_NVGPUCTRPERM-permission-issue-performance-counters
NVIDIA, DCGM feature overview: concurrent usage of NVIDIA profiling tools: https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
NVIDIA, Nsight Compute profiling guide (serialization, driver-resource reservation, DCGM conflict): https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html
NVIDIA, Nsight Systems user guide (uses CUPTI for CUDA profiling): https://docs.nvidia.com/nsight-systems/UserGuide/index.html
NVIDIA, open-gpu-kernel-modules (kernel-open/nvidia/nv-reg.h, driver 610.43.03): https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/nvidia/nv-reg.h
PyTorch, kineto (uses CUPTI to collect traces and metrics): https://github.com/pytorch/kineto/blob/main/libkineto/README.md
HPCToolkit, GPU measurement documentation (observed 2x execution time on launch-heavy code): https://hpctoolkit.gitlab.io/hpctoolkit/users/gpu/gpu.html

NVIDIA, CUPTI documentation (v2026.2.1). Note which page carries which text: the CUPTI definition, the API table, the tracing-versus-profiling definitions, "The CUPTI SDK is part of the CUDA Toolkit, and will be installed along all the other CUDA libraries", and the /usr/local/cuda-<version>/extras/CUPTI/ path are on the documentation landing page (https://docs.nvidia.com/cupti/), not on the Usage page. Verified on the Usage page: the activity-buffer callbacks and "For typical workloads, it's suggested to choose a size between 1 and 10 MB"; "CUPTI doesn't guarantee any ordering of the activities in the activity buffer as activity records for few activity kinds are added lazily"; the Correlation ID section, including "For CUPTI to generate correlationId values, the CUDA API activity kinds i.e. CUPTI_ACTIVITY_KIND_RUNTIME and/or CUPTI_ACTIVITY_KIND_DRIVER must be enabled"; the Profiling API being "not supported under MPS (Multi-Process Service), Confidential Compute, or SLI configured systems"; the framework-integration reference to "a framework with CUPTI integration (like PyTorch or TensorFlow)"; and the Multiple Subscribers section ("Starting with CUDA 13.3, CUPTI supports multiple concurrent subscribers to trace CUDA activities simultaneously", requiring "CUDA 13.3+ and driver r610+", "applies only to the CUPTI Activity API", and "CUPTI still does not support tracing or profiling along with other NVIDIA Developer Tools like cuda-gdb or Compute Sanitizer"); and the Compatibility and Requirements section ("CUPTI adheres to CUDA Toolkit compatibility requirements with CUDA drivers, which includes support for Backward, Forward and Enhanced compatibilities", "A profiling tool based on an older version of CUPTI can still operate with a more recent CUDA driver", the pointer to the CUDA Toolkit and Corresponding Driver Versions table, and "Attempting to use CUPTI calls with an incompatible CUDA driver version will result in a CUPTI_ERROR_NOT_INITIALIZED error code"). https://docs.nvidia.com/cupti/main/main.html · Internal inconsistency, flagged rather than resolved: the Troubleshooting section of the same document still describes CUPTI_ERROR_MULTIPLE_SUBSCRIBERS_NOT_SUPPORTED as meaning CUPTI "only allows one callback subscriber... to be active at a time within a process", which contradicts the Multiple Subscribers section. The Multiple Subscribers text is the newer of the two. ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩
NVIDIA, CUPTI product page: "Normally packaged with the CUDA Toolkit, NVIDIA occasionally uses this page to provide CUPTI improvements and bug fixes between toolkit releases." The feature list includes "Normalized timestamps for CPU and GPU trace". https://developer.nvidia.com/cupti ↩
NVIDIA, CUPTI release notes: CUDA 12.8 deprecated the Event API (cupti_events.h) and Metric API (cupti_metrics.h); CUDA 13.0 dropped them, and "Calling any Event or Metric API will return the error code CUPTI_ERROR_LEGACY_PROFILER_NOT_SUPPORTED. It is recommended to use the CUPTI Range Profiling API as an alternative." The PC Sampling Activity API from cupti_activity.h was also dropped in 13.0. https://docs.nvidia.com/cupti/release-notes/release-notes.html ↩↩↩
NVIDIA, CUPTI Usage, overhead: "Overhead can vary significantly from one application to another. It largely depends on the density of the CUDA activities in the application"; "In general overhead of tracing i.e. activity APIs is much lesser than the metrics profiling APIs"; serial kernel trace via CUPTI_ACTIVITY_KIND_KERNEL "can significantly change the overall performance characteristics of the application because all kernel executions are serialized on the GPU", while CUPTI_ACTIVITY_KIND_CONCURRENT_KERNEL "doesn't affect the concurrency of the kernels in the application" at the cost of code instrumentation. https://docs.nvidia.com/cupti/main/main.html ↩↩↩↩
NVIDIA, CUPTI special configurations, MIG: an isolated compute instance supports tracing and profiling; for a shared compute instance, "Due to this resource sharing, collecting profiling data from shared units is not permitted. Attempts to collect metrics from a shared unit will result in NaN values." "Tracing works for shared Compute Instances." https://docs.nvidia.com/cupti/special-configurations/special-configurations.html ↩↩
NVIDIA, ERR_NVGPUCTRPERM solution page: the error text; the restriction applying to "driver versions 419.17+ on Windows or 418.43+ on Linux"; options nvidia NVreg_RestrictProfilingToAdminUsers=0 in a .conf file under /etc/modprobe.d; the dracut --regenerate-all -f and update-initramfs -u -k all initramfs rebuilds; the RmProfilingAdminOnly flag in /proc/driver/nvidia/params; CAP_SYS_ADMIN, and "Starting in driver version R565, the CAP_PERFMON capability will also allow access"; "When profiling within a container, access must be enabled on the host, or the container must be started with the appropriate permissions by passing --cap-add=SYS_ADMIN as an admin user"; and the R610+ nvidia-capabilities system (profiler-context, profiler-device, trace-device), including "Profiling capabilities are granted system-wide and cannot be scoped to individual GPUs" and "In a future release, the regkey-based method will be removed." https://developer.nvidia.com/nvidia-development-tools-solutions-ERR_NVGPUCTRPERM-permission-issue-performance-counters ↩↩↩↩↩↩↩↩↩↩↩↩↩
NVIDIA, open-gpu-kernel-modules, kernel-open/nvidia/nv-reg.h, read at NVIDIA_VERSION = 610.43.03. The doc comment above the parameter reads "0: Do not restrict GPU counters (default)", while the definition is NV_DEFINE_REG_ENTRY(__NV_RM_PROFILING_ADMIN_ONLY_PARAMETER, 1); and the macro expands to static NvU32 __NV_REG_VAR(regkey) = (default_value);. The compiled-in default is therefore 1 (restricted) and the comment is stale. https://github.com/NVIDIA/open-gpu-kernel-modules/blob/main/kernel-open/nvidia/nv-reg.h ↩↩
NVIDIA, DCGM feature overview, Concurrent Usage of NVIDIA Profiling Tools: "Due to current hardware limitations, collection of profiling metrics with DCGM will conflict with usage of other developer tools from NVIDIA such as Nsight Systems or Nsight Compute."; the error "The requested operation could not be completed because the affected resource is in use."; dcgmi profile --pause / --resume and the dcgmProfPause() / dcgmProfResume() APIs; "When paused, DCGM will publish BLANK values for profiling metrics." https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html ↩↩↩↩↩↩
NVIDIA, Nsight Compute profiling guide: "NVIDIA Nsight Compute serializes kernel launches within the profiled application, potentially across multiple processes profiled by one or more instances of the tool at the same time."; "Serialization across processes is necessary since for the collection of HW performance metrics, some GPU and driver objects can only be acquired by a single process at a time. This is done on a per-CUDA device or MIG instance basis, meaning only one process can profile a given device at a time."; the driver-resource error names "DCGM, a client of CUPTI's Profiling API, Nsight Graphics, or another instance of NVIDIA Nsight Compute" as possible reservation holders; "If you expect the problem to be caused by DCGM, consider using dcgmi profile --pause..." Note that no NVIDIA source states Nsight Compute is itself built on CUPTI, and these passages treat a CUPTI Profiling API client as a distinct competing process; this page therefore does not claim it is. https://docs.nvidia.com/nsight-compute/ProfilingGuide/index.html ↩↩↩↩↩
NVIDIA, Nsight Systems user guide: "Nsight Systems uses CUPTI for CUDA profiling, including to collect the CUDA memory usage by the application processes." The same guide publishes a per-feature overhead figure: "Collecting Unified Memory CPU page faults can cause overhead of up to 70% in testing. Use this functionality only when needed." https://docs.nvidia.com/nsight-systems/UserGuide/index.html ↩↩
PyTorch, kineto libkineto/README.md: "NVIDIA CUPTI: used to collect traces and metrics from NVIDIA GPUs." https://github.com/pytorch/kineto/blob/main/libkineto/README.md ↩
HPCToolkit GPU documentation: "we have seen the execution time double when profiling and tracing a GPU-accelerated application that launches kernels very frequently", and, on PC sampling, "Our experience with CUPTI's serialization-based API for PC sampling is that the overhead is less than 5x." These are HPCToolkit's own measurements, not NVIDIA figures. https://hpctoolkit.gitlab.io/hpctoolkit/users/gpu/gpu.html ↩