Markdown

NVML: the library under nvidia-smi, DCGM and every GPU exporter¶

Scope: the NVIDIA Management Library as the programmatic surface you build fleet tooling on. What it exposes that shelling out to nvidia-smi does not (event-driven Xid delivery, batched field reads with per-field status, driver-buffered samples), the semantics that make its most-quoted metric misleading (utilization.gpu is a duty cycle, not a measure of how much of the GPU you are using), and the version, MIG and container traps that silently break an exporter. For the CLI, see nvidia-smi reference; for the cluster daemon layered on top of NVML, see GPU diagnostics and validation.

What it is¶

NVML is a C library that ships with the NVIDIA driver: "The NVIDIA Management Library (NVML) is a C-based programmatic interface for monitoring and managing various states within NVIDIA GPUs", and it "is also the underlying library for the NVIDIA-supported nvidia-smi tool."¹ The runtime object is libnvidia-ml.so.1, which is what NVIDIA's own Go bindings dlopen at init.⁷ It is thread-safe: "NVML is thread-safe so it is safe to make simultaneous NVML calls from multiple threads."¹

Almost everything you already run on a GPU node is an NVML client. nvidia-smi is a thin CLI over it. DCGM is layered above it: "The user space shared library, libdcgm.so.4, is the core component of DCGM. This library implements the major underlying functionality and exposes this as a set of C-based APIs. It sits on top of the NVIDIA driver, NVML, and the CUDA Toolkit."⁹ The Kubernetes device plugin, the container toolkit, and dcgm-exporter all sit on the same foundation. So NVML's semantics are not an implementation detail you can ignore: they are the semantics of every GPU number on your dashboards.

Two surfaces, with very different permission requirements:

Query, which works unprivileged: identity, utilization, ECC counters, temperature, power, clocks, running processes, remapped rows, MIG topology.
Control, which needs root: "ECC mode, ECC reset, Compute mode, Persistence mode".² nvmlDeviceSetPersistenceMode is documented "For Linux only. Requires root/admin permissions."³ The common claim that NVML needs root is wrong; only the setters do.

Bindings, and the packaging trap. The official Python binding is the PyPI package nvidia-ml-py, published by NVIDIA, and it is the package that ships the pynvml module. The separately-named pynvml package is not NVIDIA's: its own PyPI page says "This project has been deprecated. The pynvml module is NOT developed or maintained in this project!" and that it "depends on the official NVML bindings published by NVIDIA under a different nvidia-ml-py project."⁸

The behaviour differs by era, which is why the confusion persists. Up to pynvml 11.5.3 the package shipped its own pynvml module with a different API, so installing it genuinely got you a binding that did not match NVIDIA's docs. From 12.0.0 it stopped shipping that module: it now simply depends on nvidia-ml-py and installs a redirector that emits a FutureWarning pointing you at the official package. So on a current install, import pynvml resolves to NVIDIA's module either way; the hazard today is a stale pin (pynvml<12) rather than the name itself. Depend on nvidia-ml-py explicitly and the ambiguity disappears.⁸

For Go, NVIDIA/go-nvml wraps the C API rather than reimplementing it: the bindings are "not a reimplementation of NVML in Go, but rather a set of wrappers around the C API provided by libnvidia-ml.so."⁷

Why use it¶

Reach past the CLI when you are writing something that runs continuously.

Xid errors arrive as events, not as log lines you tail. nvmlEventSetWait blocks until a GPU event arrives, and for nvmlEventTypeXidCriticalError the nvmlEventData_t.eventData field "Stores Xid error for the device in the event of nvmlEventTypeXidCriticalError, eventData is 0 for any other event. eventData is set as 999 for unknown Xid error."³ That is a structured, per-device, per-MIG-instance Xid feed. Scraping dmesg for Xid strings, which is what most home-grown agents do, is a worse version of this.
Batched reads cost less than N calls. nvmlDeviceGetFieldValues states: "If any of the underlying fieldIds are populated by the same driver call, the results for those field IDs will be populated from a single call rather than making a driver call for each fieldId."³ Each returned nvmlFieldValue_t also carries latencyUsec, how long that field took to fetch, so a scraper can measure its own cost. Read it with the caveat the header attaches: it "may be averaged across several fields that are serviced by the same driver call", which is exactly the batching case, so it is a per-call cost smeared over the batch rather than a true per-field figure.
The driver already buffers samples for you. nvmlDeviceGetSamples fetches "the power, utilization or clock samples maintained in the buffer by the driver", and the documented advantage is "to get [sic] higher frequency data at lower polling cost" (the header duplicates the word).³ You advance lastSeenTimeStamp to drain only what is new.

Shelling out to nvidia-smi per scrape gets you none of this, pays process-spawn cost every interval, and forces you to parse a human-facing format that changes.

When to use it (and when not)¶

You want to	Use	Not
Read or set state on one node, by hand	`nvidia-smi`	NVML; you are writing code to do a one-liner's job
Build an exporter, scheduler plugin, or health gate	NVML	`nvidia-smi` parsing
Watch for Xid, ECC, or GPU-unavailable events	NVML event API	polling `dmesg`
Get SM occupancy, tensor-core activity, NVLink bandwidth	DCGM profiling metrics	NVML; it does not expose profiling counters
Run a diagnostic that grades the GPU	`dcgmi diag`	NVML; it has no test harness
Manage a fleet from one place	DCGM (`nv-hostengine`)	NVML; it is an in-process, single-node library

The line between NVML and DCGM is architectural, not a matter of preference: DCGM is built on NVML and adds the daemon, the health watches, the active diagnostics, and the profiling-counter path that NVML has no API for. NVIDIA publishes no "use DCGM when..." rule, so do not expect one; use the layering as the guide. If you need counters that come from the GPU's performance monitors (SM active, tensor active, DRAM bandwidth), that is CUPTI territory reached through DCGM, and it carries the exclusivity constraints that page describes. NVML never touches that hardware.

Architecture¶

flowchart TB
    DRV["NVIDIA kernel driver"] --> NVML["libnvidia-ml.so.1<br/>(NVML, ships with the driver)"]
    NVML --> SMI["nvidia-smi"]
    NVML --> DCGM["libdcgm.so.4 / nv-hostengine<br/>(health, diag, policy)"]
    NVML --> DP["k8s device plugin,<br/>container toolkit"]
    NVML --> MINE["your exporter / agent<br/>(nvidia-ml-py, go-nvml)"]
    DCGM --> EXP["dcgm-exporter"]
    DCGM -.->|"profiling counters<br/>(NOT via NVML)"| PM["GPU performance monitors"]
    EXP --> PROM["Prometheus"]
    MINE --> PROM

DCGM's dashed path is the one thing NVML cannot do. Everything else on your node comes through the same library.

How to use it: the metric that lies¶

The single most consequential thing to know about NVML is what utilization.gpu actually means. From nvml.h:

/**
 * Utilization information for a device.
 * Each sample period may be between 1 second and 1/6 second, depending on the product being queried.
 */
typedef struct nvmlUtilization_st
{
    unsigned int gpu;      //!< Percent of time over the past sample period during which one or more kernels was executing on the GPU
    unsigned int memory;   //!< Percent of time over the past sample period during which global (device) memory was being read or written
} nvmlUtilization_t;

Read it literally: gpu is the fraction of time at least one kernel was resident, not the fraction of the GPU's parallel capacity in use.³ A kernel that occupies a single SM for the whole sample period reports 100%, because for the whole period one or more kernels was executing. A well-fed GPU near peak FLOPs and a one-block kernel wasting every SM but one report the same number. Any capacity plan, autoscaler, or "our GPUs are 95% utilized" claim built on this is measuring the wrong thing. The metric you want for capacity is SM activity or tensor-core activity, which comes from the profiling counters via DCGM and CUPTI, not from NVML.

Two more documented distortions on the same field:

It is unavailable under MIG. "On MIG-enabled GPUs, querying device utilization rates is not currently supported."³ Enabling MIG silently removes your utilization telemetry, and the same is true for the GPU/memory/ENC/DEC sample types via nvmlDeviceGetSamples.
A freshly loaded driver reports garbage. "During driver initialization when ECC is enabled one can see high GPU and Memory Utilization readings. This is caused by ECC Memory Scrubbing mechanism that is performed during driver initialization."³ An exporter that starts with the driver will publish a spike that is not work. Persistence mode removes the repeated driver init that keeps recreating this.

How to develop with it: decoding the clock-event bitmask¶

The other number operators reach for is "why is this GPU slow", answered by the clock event reason bitmask from nvmlDeviceGetCurrentClocksEventReasons. Three things about it are easy to get wrong, and all three are in NVIDIA's own header.

The rename is real, and it is partial. Six reasons plus the None/All sentinels were renamed from nvmlClocksThrottleReason* to nvmlClocksEventReason*, with the old spellings retained as deprecated aliases of the same value. The three hardware reasons were not renamed at all: nvmlClocksThrottleReasonHwSlowdown (0x8), nvmlClocksThrottleReasonHwThermalSlowdown (0x40) and nvmlClocksThrottleReasonHwPowerBrakeSlowdown (0x80) keep their original names as the current, primary spelling. There is no nvmlClocksEventReasonHwSlowdown.³ A blanket search-and-replace of the identifier across your codebase will not compile. The bit values never changed, so any decoder that works on values is unaffected.

Two separate events get conflated here, so be precise about which one bit you. The nvmlClocksEventReason* macros already exist in headers shipping NVML_API_VERSION 12; what arrived with NVML_API_VERSION 13 is the DEPRECATED(13.0) attribute on the old functions, which is when the compiler started complaining.⁵ So the macros changed in one release and the warnings appeared in a later one.

A nonzero mask is not a fault. GpuIdle (0x1) is a reason. An idle GPU reports a nonzero mask, and the naive if (mask) alert() pages you at 3am because nobody is using the machine.

HwSlowdown alone is ambiguous, by NVIDIA's own admission. The header lists its causes as temperature too high, external power brake asserted, and fast-trigger power protection, and then adds two further bullets: "May be also reported during PState or clock change" and, under it, "This behavior may be removed in a later release."³ So a single sample with HwSlowdown set can just be a clock transition. NVIDIA does not document that HwSlowdown is always co-asserted with the specific thermal or power-brake bits, so do not assume it: alert on the specific bits immediately, and on bare HwSlowdown only when it persists.

The decoder below is the arithmetic that follows from those three facts, with the bit values read out of nvml.h.

"""Decoding the NVML clock-event bitmask into an alert decision.

Bit values are read from NVIDIA's nvml.h (NVML_API_VERSION 13). Note the spelling:
the three hardware reasons kept the nvmlClocksThrottleReason* name; the other six
(plus the None/All sentinels) were renamed to nvmlClocksEventReason*. Values unchanged.
"""
import numpy as np

# --- nvml.h, NVML_API_VERSION 13 ------------------------------------------
NONE                        = 0x0000000000000000  # nvmlClocksEventReasonNone
GPU_IDLE                    = 0x0000000000000001  # nvmlClocksEventReasonGpuIdle
APPLICATIONS_CLOCKS_SETTING = 0x0000000000000002  # nvmlClocksEventReasonApplicationsClocksSetting
SW_POWER_CAP                = 0x0000000000000004  # nvmlClocksEventReasonSwPowerCap
HW_SLOWDOWN                 = 0x0000000000000008  # nvmlClocksThrottleReasonHwSlowdown  (NOT renamed)
SYNC_BOOST                  = 0x0000000000000010  # nvmlClocksEventReasonSyncBoost
SW_THERMAL_SLOWDOWN         = 0x0000000000000020  # nvmlClocksEventReasonSwThermalSlowdown
HW_THERMAL_SLOWDOWN         = 0x0000000000000040  # nvmlClocksThrottleReasonHwThermalSlowdown  (NOT renamed)
HW_POWER_BRAKE_SLOWDOWN     = 0x0000000000000080  # nvmlClocksThrottleReasonHwPowerBrakeSlowdown (NOT renamed)
DISPLAY_CLOCK_SETTING       = 0x0000000000000100  # nvmlClocksEventReasonDisplayClockSetting

ALL = (GPU_IDLE | APPLICATIONS_CLOCKS_SETTING | SW_POWER_CAP | HW_SLOWDOWN | SYNC_BOOST
       | SW_THERMAL_SLOWDOWN | HW_THERMAL_SLOWDOWN | HW_POWER_BRAKE_SLOWDOWN
       | DISPLAY_CLOCK_SETTING)                    # nvmlClocksEventReasonAll
assert ALL == 0x1FF

# Severity, not a flat list. "Throttled" is not the same as "broken".
BENIGN   = GPU_IDLE                                                  # no work to do
OPERATOR = APPLICATIONS_CLOCKS_SETTING | DISPLAY_CLOCK_SETTING | SYNC_BOOST   # someone set this
DERATE   = SW_POWER_CAP | SW_THERMAL_SLOWDOWN                        # firmware is managing us
FAULT    = HW_THERMAL_SLOWDOWN | HW_POWER_BRAKE_SLOWDOWN             # hardware protection fired

NAMES = {GPU_IDLE: "GpuIdle", APPLICATIONS_CLOCKS_SETTING: "ApplicationsClocksSetting",
         SW_POWER_CAP: "SwPowerCap", HW_SLOWDOWN: "HwSlowdown", SYNC_BOOST: "SyncBoost",
         SW_THERMAL_SLOWDOWN: "SwThermalSlowdown", HW_THERMAL_SLOWDOWN: "HwThermalSlowdown",
         HW_POWER_BRAKE_SLOWDOWN: "HwPowerBrakeSlowdown",
         DISPLAY_CLOCK_SETTING: "DisplayClockSetting"}

def decode(mask):
    """Bits set, plus any bit NVML added that this build does not know about."""
    known = sorted(b for b in NAMES if mask & b)
    return [NAMES[b] for b in known], mask & ~ALL      # (names, unknown_bits)

def classify(mask):
    """Severity of a single sample. HwSlowdown alone is deliberately not a fault."""
    _, unknown = decode(mask)
    if unknown:
        return "unknown"                # a future NVML bit: surface it, never drop it
    if mask & FAULT:
        return "fault"
    if mask & DERATE:
        return "derate"
    if mask & HW_SLOWDOWN:
        return "ambiguous"              # see the nvml.h note: may also fire on a PState change
    if mask & OPERATOR:
        return "operator"
    if mask & BENIGN:
        return "benign"
    return "clear"

def alert(samples, sustained=3):
    """Page only on a fault, or on HwSlowdown that persists across `sustained` samples."""
    cls = [classify(m) for m in samples]
    if "fault" in cls:
        return True
    run = 0
    for c in cls:
        run = run + 1 if c == "ambiguous" else 0
        if run >= sustained:
            return True
    return False

# --- Happy path: a healthy busy GPU reports nothing ------------------------
assert decode(NONE) == ([], 0)
assert classify(NONE) == "clear"
print(f"mask 0x{NONE:03x}: {classify(NONE)}")

# --- Adversarial 1: an IDLE GPU is "throttled". Any nonzero-means-bad rule pages on it.
names, _ = decode(GPU_IDLE)
assert names == ["GpuIdle"]
assert GPU_IDLE != 0                       # a naive `if mask: alert()` fires here
assert classify(GPU_IDLE) == "benign"
assert alert([GPU_IDLE] * 100) is False    # never page, however long it persists
print(f"mask 0x{GPU_IDLE:03x}: {classify(GPU_IDLE)} ({names[0]}), a nonzero mask that must not page")

# --- Adversarial 2: real thermal protection, both bits set -----------------
mask = HW_SLOWDOWN | HW_THERMAL_SLOWDOWN
assert mask == 0x48
names, unknown = decode(mask)
assert names == ["HwSlowdown", "HwThermalSlowdown"] and unknown == 0
assert classify(mask) == "fault"
assert alert([mask]) is True               # one sample is enough: hardware protection fired
print(f"mask 0x{mask:03x}: {classify(mask)} ({', '.join(names)}), pages on the first sample")

# --- Adversarial 3: HwSlowdown ALONE is ambiguous by NVIDIA's own note -----
# nvml.h: HwSlowdown "May be also reported during PState or clock change".
assert classify(HW_SLOWDOWN) == "ambiguous"
assert alert([HW_SLOWDOWN, NONE, HW_SLOWDOWN]) is False   # a blip during a clock change
assert alert([HW_SLOWDOWN] * 3) is True                   # sustained: real derating
print(f"mask 0x{HW_SLOWDOWN:03x}: {classify(HW_SLOWDOWN)}; blip -> no page, 3 in a row -> page")

# --- Adversarial 4: a future NVML bit must not be silently dropped ---------
# nvml.h on nvmlClocksEventReasonAll: "New reasons might be added to this list in the future".
future = 0x0000000000000200
names, unknown = decode(SW_POWER_CAP | future)
assert names == ["SwPowerCap"]             # the known bit still decodes
assert unknown == future                   # but the unknown one is surfaced, not lost
assert classify(SW_POWER_CAP | future) == "unknown"
print(f"mask 0x{SW_POWER_CAP | future:03x}: known={names}, unknown bits=0x{unknown:03x} (surfaced, not dropped)")

# --- Adversarial 5: the rename is PARTIAL, and value-identical -------------
# nvml.h declares 6 reasons (plus the None sentinel) as nvmlClocksEventReason*, each with
# the old nvmlClocksThrottleReason* name kept as a deprecated alias of the SAME value, and
# 3 hardware reasons never renamed at all. Decoding by value is safe; a blanket sed of the
# identifier ClocksThrottleReason -> ClocksEventReason does not compile.
EVENT_SPELLING = {"GpuIdle": 0x1, "ApplicationsClocksSetting": 0x2, "SwPowerCap": 0x4,
                  "SyncBoost": 0x10, "SwThermalSlowdown": 0x20,
                  "DisplayClockSetting": 0x100, "None": 0x0}
THROTTLE_ONLY = {"HwSlowdown": 0x8, "HwThermalSlowdown": 0x40, "HwPowerBrakeSlowdown": 0x80}

# The deprecated ThrottleReason alias of each renamed reason is #defined to the SAME
# value as its EventReason primary, so a mask built from either spelling decodes alike.
DEPRECATED_ALIAS = {"GpuIdle": 0x1, "ApplicationsClocksSetting": 0x2, "SwPowerCap": 0x4,
                    "SyncBoost": 0x10, "SwThermalSlowdown": 0x20,
                    "DisplayClockSetting": 0x100, "None": 0x0}   # nvmlClocksThrottleReason*
assert DEPRECATED_ALIAS == EVENT_SPELLING                  # same names -> same bit values
for reason, value in EVENT_SPELLING.items():
    assert decode(value) == decode(DEPRECATED_ALIAS[reason])   # alias decodes identically
assert set(EVENT_SPELLING) & set(THROTTLE_ONLY) == set()   # no reason carries both spellings
assert all(bit & ALL for bit in THROTTLE_ONLY.values())    # yet all 3 HW bits are in ALL
assert not any(name.startswith("Hw") for name in EVENT_SPELLING)  # no EventReasonHw* exists
mixed = EVENT_SPELLING["SwPowerCap"] | THROTTLE_ONLY["HwSlowdown"]
assert decode(mixed)[0] == ["SwPowerCap", "HwSlowdown"]    # both spellings in one mask
assert sum(EVENT_SPELLING.values()) | sum(THROTTLE_ONLY.values()) == ALL
print(f"{len(EVENT_SPELLING) - 1} reasons renamed to ClocksEventReason*, "
      f"{len(THROTTLE_ONLY)} kept ClocksThrottleReason*; all 9 bits still in ALL=0x{ALL:03x}")

# --- Adversarial 6: a scrape window, decoded in bulk -----------------------
window = np.array([NONE, NONE, SW_POWER_CAP, SW_POWER_CAP | HW_SLOWDOWN,
                   HW_SLOWDOWN | HW_THERMAL_SLOWDOWN, NONE], dtype=np.uint64)
kinds = [classify(int(m)) for m in window]
assert kinds == ["clear", "clear", "derate", "derate", "fault", "clear"]
assert alert(window.tolist()) is True
busy = np.count_nonzero(window & np.uint64(~ALL & 0xFFFFFFFFFFFFFFFF))
assert busy == 0                           # no unknown bits anywhere in the window
print(f"6-sample window -> {kinds}, page={alert(window.tolist())}")

Executed output:

mask 0x000: clear
mask 0x001: benign (GpuIdle), a nonzero mask that must not page
mask 0x048: fault (HwSlowdown, HwThermalSlowdown), pages on the first sample
mask 0x008: ambiguous; blip -> no page, 3 in a row -> page
mask 0x204: known=['SwPowerCap'], unknown bits=0x200 (surfaced, not dropped)
6 reasons renamed to ClocksEventReason*, 3 kept ClocksThrottleReason*; all 9 bits still in ALL=0x1ff
6-sample window -> ['clear', 'clear', 'derate', 'derate', 'fault', 'clear'], page=True

The unknown path is the part most decoders skip. The header says of the All mask: "New reasons might be added to this list in the future". A decoder that iterates its known bits and drops the rest will keep reporting "clear" on a GPU that a newer driver is telling it something new about. Mask with ~ALL and surface the remainder.

How to maintain it: versions, deprecations, and the ABI¶

NVML versions individual functions and structs rather than the library. The header sets NVML_API_VERSION 13 and, by default, maps unsuffixed names onto the newest implementation:

#ifndef NVML_NO_UNVERSIONED_FUNC_DEFS
    #define nvmlInit                             nvmlInit_v2
    #define nvmlDeviceGetPciInfo                 nvmlDeviceGetPciInfo_v3
    #define nvmlDeviceGetComputeRunningProcesses nvmlDeviceGetComputeRunningProcesses_v3
    #define nvmlEventSetWait                     nvmlEventSetWait_v2
    // ... 16 more, elided
#endif

Define NVML_NO_UNVERSIONED_FUNC_DEFS and the unsuffixed aliases go away, so you must name a version explicitly (nvmlInit_v2). The header states no rationale for the guard, and note it does not by itself pin you to an older surface; it forces you to say which one you mean.³ The change log describes the mechanism: "Functions that changed API and/or size of structs have appended versioning suffix (e.g., nvmlDeviceGetPciInfo_v2). Appropriate C defines have been added that map old function names to the newer version of the function."⁴

Be careful about how strong a guarantee you infer from that. NVIDIA's own Go bindings assert it ("Since the NVML API is guaranteed to be backwards compatible, we should strive to keep this always up to date with the latest"),⁷ but the NVML documentation itself never states such a guarantee (it describes the mechanism, not a promise), and its Known Issues page documents a real break: field values numbered 251 to 273 "have changed between 13.0 and 13.0U1/v580TRD2."⁶ The versioned function and struct surface is stable. Numeric field IDs are not a contract in the same way.

A batch of functions carry DEPRECATED(13.0) in the current header, which on Linux expands to a compiler warning:³

Deprecated in 13.0	Replacement
`nvmlDeviceGetCurrentClocksThrottleReasons`	`nvmlDeviceGetCurrentClocksEventReasons`
`nvmlDeviceGetSupportedClocksThrottleReasons`	`nvmlDeviceGetSupportedClocksEventReasons`
`nvmlDeviceGetTemperature`	`nvmlDeviceGetTemperatureV`
`nvmlDeviceGetPowerState`	`nvmlDeviceGetPerformanceState` ("This function exposes an incorrect generalization")
`nvmlDeviceGetHandleBySerial`	`nvmlDeviceGetHandleByUUID`
`nvmlDeviceGetApplicationsClock`	None: "Applications clocks are deprecated and will be removed in CUDA 14.0"

nvmlDeviceGetTemperature is the one that will surprise an exporter maintainer: the function every GPU exporter has called since 2012 is now deprecated in favour of nvmlDeviceGetTemperatureV.

How to run it in production¶

Persistence mode changes what your ECC counters mean. This is not a footnote, it is the difference between a usable and a useless signal. From the header, above the ECC counter enum: "Volatile counts are reset each time the driver loads. On Windows this is once per boot. On Linux this can be more frequent. On Linux the driver unloads when no active clients exist. If persistence mode is enabled or there is always a driver client active (e.g. X11), then Linux also sees per-boot behavior. If not, volatile counts are reset each time a compute app is run."³ On a node without persistence mode, the volatile ECC counter resets between jobs, so an exporter reading it is reporting "errors since the last job started", which is not what any dashboard label claims. Aggregate counts "persist across reboots (i.e. for the lifetime of the device)".³

Alert on the right row-remap field. nvmlDeviceGetRemappedRows returns isPending and failureOccurred. The header is explicit that pending is benign: "A pending remapping won't affect future work on the GPU since error-containment and dynamic page blacklisting will take care of that." failureOccurred "will be set if a row remapping ever failed in the past".³ That is the RMA-grade signal; isPending merely means a reset is needed. (The header states that a remapping failed, and stops there. The usual reading, that the device has run out of spare rows, is an interpretation this page is making, not a sentence NVML prints.) Also note it is unavailable under MIG with active instances.

Check the per-field return code, not just the call. nvmlDeviceGetFieldValues returns NVML_SUCCESS "if any values in values were populated", and each nvmlFieldValue_t carries its own nvmlReturn which "must be checked before looking at value, as value is undefined if nvmlReturn != NVML_SUCCESS".³ An exporter that checks only the top-level status will happily publish uninitialized memory as a metric for every field the device does not support.

The Xid event loop's normal path is a timeout. nvmlEventSetWait returns NVML_ERROR_TIMEOUT when no event arrived, and the docs warn it "in certain conditions can return before specified timeout passes (e.g. when interrupt arrives)".³ Treat timeout as the steady state, not an error. On Linux "every Xid error event would return the associated event data"; on Windows the API coalesces to the last-seen Xid.³

Containers: the NVML failure you will actually hit. The documented symptom is Failed to initialize NVML: Unknown Error, and the cause is not NVML. The NVIDIA Container Toolkit's runtime hook "makes modifications, including setting up cgroup access, to the container without the low-level runtime being aware of these changes", so a later container update (classically a systemctl daemon-reload) strips the GPU device access from a running container and NVML can no longer initialize.¹⁰ The documented fixes are to set Docker's cgroup driver to cgroupfs ("exec-opts": ["native.cgroupdriver=cgroupfs"]), to pass the device nodes explicitly, or to use CDI, where "the required device nodes are included in the modifications made to the container config".¹⁰ See container toolkit.

Failure modes¶

Symptom	Cause	Fix
Dashboards say "GPUs are 95% utilized" but throughput is poor	`utilization.gpu` is percent of time a kernel was resident, not capacity used³	Measure SM/tensor activity via DCGM profiling metrics; treat NVML utilization as a liveness signal only
GPU utilization disappeared after enabling MIG	"On MIG-enabled GPUs, querying device utilization rates is not currently supported."³	Use per-instance DCGM metrics; do not expect NVML utilization under MIG
A utilization spike at driver load with no job running	ECC memory scrubbing during driver init³	Enable persistence mode; ignore the first samples after driver load
Volatile ECC counters keep resetting to zero	Without persistence mode, the driver unloads when the last client exits and volatile counts reset per compute app³	Enable persistence mode; alert on aggregate counters
Pager fires on an idle GPU	`GpuIdle` is a nonzero clock-event reason (executed model above)	Classify the mask by severity; never treat nonzero as fault
Pager fires whenever clocks change	Bare `HwSlowdown` "May be also reported during PState or clock change"³	Require the specific thermal/power-brake bits, or a sustained run of `HwSlowdown`
The exporter publishes plausible garbage for unsupported fields	Only the top-level status was checked; per-field `nvmlReturn` was not³	Check `value.nvmlReturn` per field before reading `value.value`
Code stops compiling after a driver update	`nvmlDeviceGetCurrentClocksThrottleReasons`, `nvmlDeviceGetTemperature` and others are `DEPRECATED(13.0)`³	Move to the `EventReasons` and `TemperatureV` entry points; values are unchanged
A blanket rename of `ClocksThrottleReason` broke the build	The three hardware reasons were never renamed³	Keep `nvmlClocksThrottleReasonHwSlowdown`, `...HwThermalSlowdown`, `...HwPowerBrakeSlowdown`
`Failed to initialize NVML: Unknown Error` in a running container	Container update stripped cgroup device access set up by the runtime hook¹⁰	`native.cgroupdriver=cgroupfs`, explicit `--device`, or CDI injection
`import pynvml` gives an API that does not match the docs	A stale pin of the `pynvml` package at `<12`, which shipped its own divergent module⁸	Depend on `nvidia-ml-py`; from `pynvml` 12.0.0 the package is only a redirector onto it and emits a `FutureWarning`

References¶

NVIDIA, NVML API Reference Guide (vR610, last updated 2026-05-26): https://docs.nvidia.com/deploy/nvml-api/
NVIDIA, NVIDIA Management Library (NVML) product page: https://developer.nvidia.com/nvidia-management-library-nvml
NVIDIA, NVML change log: https://docs.nvidia.com/deploy/nvml-api/change-log.html
NVIDIA, NVML known issues: https://docs.nvidia.com/deploy/nvml-api/known-issues.html
NVIDIA, nvml.h as vendored in NVIDIA/go-nvml (NVML_API_VERSION 13; go-nvml states it "is a direct copy of nvml.h from the NVIDIA driver"): https://github.com/NVIDIA/go-nvml/blob/main/gen/nvml/nvml.h
NVIDIA, go-nvml: https://github.com/NVIDIA/go-nvml
NVIDIA, nvidia-ml-py (the official Python binding): https://pypi.org/project/nvidia-ml-py/
pynvml on PyPI (deprecated; not the NVML team's binding): https://pypi.org/project/pynvml/
NVIDIA, DCGM user guide, Getting Started (DCGM sits on top of the driver, NVML and the CUDA Toolkit): https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html
NVIDIA, Container Toolkit troubleshooting (Failed to initialize NVML: Unknown Error): https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/troubleshooting.html

NVIDIA, NVML API Reference (vR610): "The NVIDIA Management Library (NVML) is a C-based programmatic interface for monitoring and managing various states within NVIDIA GPUs." "...intended to be a platform for building 3rd party applications, and is also the underlying library for the NVIDIA-supported nvidia-smi tool." "NVML is thread-safe so it is safe to make simultaneous NVML calls from multiple threads." https://docs.nvidia.com/deploy/nvml-api/nvml-api-reference.html · The product page scopes NVML to "Data Center GPUs" while the API Reference says "NVIDIA GPUs"; the API Reference is the accurate one, since NVML loads on consumer parts with many calls returning NVML_ERROR_NOT_SUPPORTED. ↩↩
NVIDIA, NVML product page: modifiable state is "ECC mode, ECC reset, Compute mode, Persistence mode"; "The NVIDIA Management Library can be downloaded as part of the NVIDIA GPU Driver for Linux and Windows." https://developer.nvidia.com/nvidia-management-library-nvml ↩
nvml.h, NVML_API_VERSION 13, read directly from the copy NVIDIA vendors into go-nvml (whose README states the file "is a direct copy of nvml.h from the NVIDIA driver"). Verified in this file: the nvmlUtilization_t field comments and the "Each sample period may be between 1 second and 1/6 second" note; the nvmlDeviceGetUtilizationRates MIG and ECC-scrubbing notes; the clock-event reason bit values, with nvmlClocksThrottleReasonHwSlowdown = 0x8, nvmlClocksThrottleReasonHwThermalSlowdown = 0x40 and nvmlClocksThrottleReasonHwPowerBrakeSlowdown = 0x80 retained under the ThrottleReason spelling while the other six reasons plus None/All are primarily nvmlClocksEventReason* with deprecated same-value aliases; the HwSlowdown cause list including "May be also reported during PState or clock change"; "New reasons might be added to this list in the future"; nvmlEventData_t.eventData carrying the Xid (0 for other events, 999 for unknown); nvmlEventSetWait timeout and Linux/Windows Xid delivery semantics; the nvmlEccCounterType_t volatile-vs-aggregate comment; nvmlDeviceGetRemappedRows isPending / failureOccurred semantics; nvmlDeviceGetFieldValues batching and the per-field nvmlReturn requirement; nvmlDeviceGetSamples driver buffering; the NVML_NO_UNVERSIONED_FUNC_DEFS guard and the _vN mapping block; and the DEPRECATED(13.0) markers on nvmlDeviceGetCurrentClocksThrottleReasons, nvmlDeviceGetSupportedClocksThrottleReasons, nvmlDeviceGetTemperature (replaced by nvmlDeviceGetTemperatureV), nvmlDeviceGetPowerState, nvmlDeviceGetHandleBySerial and nvmlDeviceGetApplicationsClock. https://github.com/NVIDIA/go-nvml/blob/main/gen/nvml/nvml.h ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩
NVIDIA, NVML change log: "Functions that changed API and/or size of structs have appended versioning suffix (e.g., nvmlDeviceGetPciInfo_v2). Appropriate C defines have been added that map old function names to the newer version of the function." The change log also records, without the page pinning a driver branch to it: "Introduced ClockEventReasons and related APIs which should be used instead of ClockThrottleReasons. Deprecated ClockThrottleReasons." https://docs.nvidia.com/deploy/nvml-api/change-log.html ↩
Verified by diffing the nvml.h that NVIDIA vendors into go-nvml across release tags: at v0.12.4-0 the header declares NVML_API_VERSION 12 and already defines nvmlClocksEventReasonGpuIdle, while v0.12.1-0 (also API version 12) does not; the DEPRECATED(13.0) markers on nvmlDeviceGetCurrentClocksThrottleReasons and friends appear only from v0.13.0-0 (NVML_API_VERSION 13). So the macro rename and the function deprecation landed in different releases. https://github.com/NVIDIA/go-nvml/blob/main/gen/nvml/nvml.h ↩
NVIDIA, NVML known issues (vR610): "NVML Field Values from #251 - #273 (Power Smoothing, Clock Event Reason, and Sync Power Balancing related field values) have changed between 13.0 and 13.0U1/v580TRD2." https://docs.nvidia.com/deploy/nvml-api/known-issues.html ↩
NVIDIA, go-nvml README: "These bindings are not a reimplementation of NVML in Go, but rather a set of wrappers around the C API provided by libnvidia-ml.so." "A working NVIDIA driver with libnvidia-ml.so is not required to compile code that imports these bindings. However, you will get a runtime error if libnvidia-ml.so is not available in your library path at runtime." "The nvml.h file is a direct copy of nvml.h from the NVIDIA driver. Since the NVML API is guaranteed to be backwards compatible, we should strive to keep this always up to date with the latest." The loader in pkg/nvml/lib.go sets defaultNvmlLibraryName = "libnvidia-ml.so.1". https://github.com/NVIDIA/go-nvml ↩↩↩
PyPI, pynvml: "This project has been deprecated. The pynvml module is NOT developed or maintained in this project! This project provides unofficial NVML Python utilities (i.e. the pynvml_utils module)." "The pynvml_utils module depends on the official NVML bindings published by NVIDIA under a different nvidia-ml-py project." NVIDIA's own binding is nvidia-ml-py, published by NVIDIA Corporation, and it is the distribution that ships the pynvml.py module. Wheel contents show the split by era: pynvml 11.5.3 ships its own pynvml/ package, while pynvml 12.0.0 and later ship only pynvml_utils/ plus a _pynvml_redirector and declare a dependency on nvidia-ml-py. https://pypi.org/project/pynvml/ · https://pypi.org/project/nvidia-ml-py/ ↩↩↩
NVIDIA, DCGM user guide: "The user space shared library, libdcgm.so.4, is the core component of DCGM. This library implements the major underlying functionality and exposes this as a set of C-based APIs. It sits on top of the NVIDIA driver, NVML, and the CUDA Toolkit." NVIDIA publishes no explicit "use DCGM instead of NVML when..." rule; the layering is the only documented relationship. https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html ↩
NVIDIA, Container Toolkit troubleshooting: the Failed to initialize NVML: Unknown Error symptom, its cause (the runtime hook "makes modifications, including setting up cgroup access, to the container without the low-level runtime being aware of these changes", so a container update such as a systemctl daemon-reload removes GPU access), and the three documented mitigations (native.cgroupdriver=cgroupfs, explicit device nodes, or CDI, where "the required device nodes are included in the modifications made to the container config"). https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/troubleshooting.html ↩↩↩