NVML: the library under nvidia-smi, DCGM and every GPU exporter¶
Scope: the NVIDIA Management Library as the programmatic surface you build fleet tooling on, including a minimal exporter (init/shutdown, device enumeration, batched field reads with per-field status, Xid event registration and wait, Prometheus rendering, and driver-restart recovery) built and executed against nvidia-ml-py==13.610.43. What it exposes that shelling out to nvidia-smi does not (event-driven Xid delivery, batched field reads with per-field status, driver-buffered samples), the semantics that make its most-quoted metric misleading (utilization.gpu is a duty cycle, not a measure of how much of the GPU you are using), and the version, MIG and container traps that silently break an exporter. For the CLI, see nvidia-smi reference; for the cluster daemon layered on top of NVML, see GPU diagnostics and validation.
Executed here: a clean virtualenv (Python 3.12.3),
pip install nvidia-ml-py==13.610.43(NVIDIA's own PyPI binding; the only third-party import the exporter below needs), against a host with no bound NVIDIA kernel driver and nonvidia-smi. Every call the exporter makes (nvmlInit,nvmlShutdown,nvmlDeviceGetCount,nvmlDeviceGetHandleByIndex,nvmlDeviceGetFieldValues,nvmlEventSetCreate/nvmlDeviceRegisterEvents/nvmlEventSetWait/nvmlEventSetFree) is the real library function, not a stand-in, and running it here produced two distinct, real NVML initialization failures at different points in the same session, reproduced verbatim below, because this sandbox genuinely has no working GPU driver stack. Per-field error handling, timeout-as-normal semantics, and the Xid event-registration failure path are exercised by constructing the real ctypes structures (c_nvmlFieldValue_t) and real exception classes (NVMLError_Timeout,NVMLError_GpuIsLost) the binding itself defines, since no device exists here to populate them through a live call. The one place a library call is monkeypatched (the driver-restart-recovery test) is labelled at the call site and tests only this page's own reconnect logic against a real, documented NVML error code, not a fabricated one. MIG-mode behavior is documented from NVIDIA's header and change log only, since exercising it needs physical Ampere-or-newer hardware this sandbox does not have.
What it is¶
NVML is a C library that ships with the NVIDIA driver: "The NVIDIA Management Library (NVML) is a C-based programmatic interface for monitoring and managing various states within NVIDIA GPUs", and it "is also the underlying library for the NVIDIA-supported nvidia-smi tool."1 The runtime object is libnvidia-ml.so.1, which is what NVIDIA's own Go bindings dlopen at init.7 It is thread-safe: "NVML is thread-safe so it is safe to make simultaneous NVML calls from multiple threads."1
Almost everything you already run on a GPU node is an NVML client. nvidia-smi is a thin CLI over it. DCGM is layered above it: "The user space shared library, libdcgm.so.4, is the core component of DCGM. This library implements the major underlying functionality and exposes this as a set of C-based APIs. It sits on top of the NVIDIA driver, NVML, and the CUDA Toolkit."9 The Kubernetes device plugin, the container toolkit, and dcgm-exporter all sit on the same foundation. So NVML's semantics are not an implementation detail you can ignore: they are the semantics of every GPU number on your dashboards.
Two surfaces, with very different permission requirements:
- Query, which works unprivileged: identity, utilization, ECC counters, temperature, power, clocks, running processes, remapped rows, MIG topology.
- Control, which needs root: "ECC mode, ECC reset, Compute mode, Persistence mode".2
nvmlDeviceSetPersistenceModeis documented "For Linux only. Requires root/admin permissions."3 The common claim that NVML needs root is wrong; only the setters do.
Bindings, and the packaging trap. The official Python binding is the PyPI package nvidia-ml-py, published by NVIDIA, and it is the package that ships the pynvml module. The separately-named pynvml package is not NVIDIA's: its own PyPI page says "This project has been deprecated. The pynvml module is NOT developed or maintained in this project!" and that it "depends on the official NVML bindings published by NVIDIA under a different nvidia-ml-py project."8
The behaviour differs by era, which is why the confusion persists. Up to pynvml 11.5.3 the package shipped its own pynvml module with a different API, so installing it genuinely got you a binding that did not match NVIDIA's docs. From 12.0.0 it stopped shipping that module: it now simply depends on nvidia-ml-py and installs a redirector that emits a FutureWarning pointing you at the official package. So on a current install, import pynvml resolves to NVIDIA's module either way; the hazard today is a stale pin (pynvml<12) rather than the name itself. Depend on nvidia-ml-py explicitly and the ambiguity disappears.8
For Go, NVIDIA/go-nvml wraps the C API rather than reimplementing it: the bindings are "not a reimplementation of NVML in Go, but rather a set of wrappers around the C API provided by libnvidia-ml.so."7
Why use it¶
Reach past the CLI when you are writing something that runs continuously.
- Xid errors arrive as events, not as log lines you tail.
nvmlEventSetWaitblocks until a GPU event arrives, and fornvmlEventTypeXidCriticalErrorthenvmlEventData_t.eventDatafield "Stores Xid error for the device in the event of nvmlEventTypeXidCriticalError, eventData is 0 for any other event. eventData is set as 999 for unknown Xid error."3 That is a structured, per-device, per-MIG-instance Xid feed. Scrapingdmesgfor Xid strings, which is what most home-grown agents do, is a worse version of this. - Batched reads cost less than N calls.
nvmlDeviceGetFieldValuesstates: "If any of the underlying fieldIds are populated by the same driver call, the results for those field IDs will be populated from a single call rather than making a driver call for each fieldId."3 Each returnednvmlFieldValue_talso carrieslatencyUsec, how long that field took to fetch, so a scraper can measure its own cost. Read it with the caveat the header attaches: it "may be averaged across several fields that are serviced by the same driver call", which is exactly the batching case, so it is a per-call cost smeared over the batch rather than a true per-field figure. - The driver already buffers samples for you.
nvmlDeviceGetSamplesfetches "the power, utilization or clock samples maintained in the buffer by the driver", and the documented advantage is "to get [sic] higher frequency data at lower polling cost" (the header duplicates the word).3 You advancelastSeenTimeStampto drain only what is new.
Shelling out to nvidia-smi per scrape gets you none of this, pays process-spawn cost every interval, and forces you to parse a human-facing format that changes.
When to use it (and when not)¶
| You want to | Use | Not |
|---|---|---|
| Read or set state on one node, by hand | nvidia-smi |
NVML; you are writing code to do a one-liner's job |
| Build an exporter, scheduler plugin, or health gate | NVML | nvidia-smi parsing |
| Watch for Xid, ECC, or GPU-unavailable events | NVML event API | polling dmesg |
| Get SM occupancy, tensor-core activity, NVLink bandwidth | DCGM profiling metrics | NVML; it does not expose profiling counters |
| Run a diagnostic that grades the GPU | dcgmi diag |
NVML; it has no test harness |
| Manage a fleet from one place | DCGM (nv-hostengine) |
NVML; it is an in-process, single-node library |
The line between NVML and DCGM is architectural, not a matter of preference: DCGM is built on NVML and adds the daemon, the health watches, the active diagnostics, and the profiling-counter path that NVML has no API for. NVIDIA publishes no "use DCGM when..." rule, so do not expect one; use the layering as the guide. If you need counters that come from the GPU's performance monitors (SM active, tensor active, DRAM bandwidth), that is CUPTI territory reached through DCGM, and it carries the exclusivity constraints that page describes. NVML never touches that hardware.
Architecture¶
flowchart TB
DRV["NVIDIA kernel driver"] --> NVML["libnvidia-ml.so.1<br/>(NVML, ships with the driver)"]
NVML --> SMI["nvidia-smi"]
NVML --> DCGM["libdcgm.so.4 / nv-hostengine<br/>(health, diag, policy)"]
NVML --> DP["k8s device plugin,<br/>container toolkit"]
NVML --> MINE["your exporter / agent<br/>(nvidia-ml-py, go-nvml)"]
DCGM --> EXP["dcgm-exporter"]
DCGM -.->|"profiling counters<br/>(NOT via NVML)"| PM["GPU performance monitors"]
EXP --> PROM["Prometheus"]
MINE --> PROMDCGM's dashed path is the one thing NVML cannot do. Everything else on your node comes through the same library.
How to use it: environment, the init/shutdown lifecycle, and the metric that lies¶
A pinned environment and the real init/shutdown lifecycle¶
The official Python binding is a single file, pynvml.py, shipped inside the nvidia-ml-py wheel; it ctypes-loads libnvidia-ml.so.1 on first use and does nothing else at import time, so pinning it is a one-line install:
Collecting nvidia-ml-py==13.610.43
Using cached nvidia_ml_py-13.610.43-py3-none-any.whl.metadata (9.7 kB)
Using cached nvidia_ml_py-13.610.43-py3-none-any.whl (53 kB)
Installing collected packages: nvidia-ml-py
Successfully installed nvidia-ml-py-13.610.43
Every NVML session has the same shape: nvmlInit() once, nvmlDeviceGetHandleByIndex() per device, work, then nvmlShutdown() once. nvmlInit_v2's own doc comment states the library "maintain[s]" a "reference count of the number of initializations. Shutdown only occurs when the reference count reaches zero", so independent callers in the same process can each init/shutdown without disturbing one another.3
This sandbox has no bound NVIDIA kernel driver and no nvidia-smi, so nvmlInit() was run for real, twice, at different points while writing this page, and failed both times with different, real, documented NVML errors. First, with no libnvidia-ml.so.1 anywhere on the library search path at all:
>>> import pynvml
>>> pynvml.nvmlInit()
Traceback (most recent call last):
File ".../pynvml.py", line 3019, in _LoadNvmlLibrary
nvmlLib = CDLL("libnvidia-ml.so.1")
File "/usr/lib/python3.12/ctypes/__init__.py", line 379, in __init__
self._handle = _dlopen(self._name, mode)
OSError: libnvidia-ml.so.1: cannot open shared object file: No such file or directory
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "<string>", line 3, in <module>
File ".../pynvml.py", line 2991, in nvmlInit
nvmlInitWithFlags(0)
File ".../pynvml.py", line 2974, in nvmlInitWithFlags
_LoadNvmlLibrary()
File ".../pynvml.py", line 3021, in _LoadNvmlLibrary
_nvmlCheckReturn(NVML_ERROR_LIBRARY_NOT_FOUND)
File ".../pynvml.py", line 1098, in _nvmlCheckReturn
raise NVMLError(ret)
pynvml.NVMLError_LibraryNotFound: NVML Shared Library Not Found
Partway through writing this page, an unrelated background package install on this host (apt-get install -y nvidia-cuda-toolkit, requested independently of this exporter and of anything in this KB) pulled in libnvidia-compute-535 as a dependency, which ships a real libnvidia-ml.so.1 userspace stub with no kernel module behind it. From that point on, the identical call fails differently:
>>> import pynvml
>>> pynvml.nvmlInit()
Traceback (most recent call last):
File "<string>", line 1, in <module>
File ".../pynvml.py", line 2991, in nvmlInit
nvmlInitWithFlags(0)
File ".../pynvml.py", line 2981, in nvmlInitWithFlags
_nvmlCheckReturn(ret)
File ".../pynvml.py", line 1098, in _nvmlCheckReturn
raise NVMLError(ret)
pynvml.NVMLError_DriverNotLoaded: Driver Not Loaded
Both are real, both are documented in nvml.h's return-code list for nvmlInit_v2 (NVML_ERROR_LIBRARY_NOT_FOUND, and NVML_ERROR_DRIVER_NOT_LOADED "if NVIDIA driver is not running"),3 and they are not the same failure. The first means the shared object could not be found at all (no driver package present anywhere on the host). The second means the shared object loaded but the kernel module behind it is not bound, which is exactly the state a container can be left in after a host driver upgrade or a systemctl daemon-reload that strips device access.10 An exporter that reports both as one generic "NVML init failed" string throws away the distinction between "install the driver" and "the driver is there but not attached to this GPU," which are different runbook branches; nvml_connect() in the exporter below catches each NVML exception class separately for exactly this reason.
The metric that lies¶
The single most consequential thing to know about NVML is what utilization.gpu actually means. From nvml.h:
/**
* Utilization information for a device.
* Each sample period may be between 1 second and 1/6 second, depending on the product being queried.
*/
typedef struct nvmlUtilization_st
{
unsigned int gpu; //!< Percent of time over the past sample period during which one or more kernels was executing on the GPU
unsigned int memory; //!< Percent of time over the past sample period during which global (device) memory was being read or written
} nvmlUtilization_t;
Read it literally: gpu is the fraction of time at least one kernel was resident, not the fraction of the GPU's parallel capacity in use.3 A kernel that occupies a single SM for the whole sample period reports 100%, because for the whole period one or more kernels was executing. A well-fed GPU near peak FLOPs and a one-block kernel wasting every SM but one report the same number. Any capacity plan, autoscaler, or "our GPUs are 95% utilized" claim built on this is measuring the wrong thing. The metric you want for capacity is SM activity or tensor-core activity, which comes from the profiling counters via DCGM and CUPTI, not from NVML.
Two more documented distortions on the same field:
- It is unavailable under MIG. "On MIG-enabled GPUs, querying device utilization rates is not currently supported."3 Enabling MIG silently removes your utilization telemetry, and the same is true for the GPU/memory/ENC/DEC sample types via
nvmlDeviceGetSamples. - A freshly loaded driver reports garbage. "During driver initialization when ECC is enabled one can see high GPU and Memory Utilization readings. This is caused by ECC Memory Scrubbing mechanism that is performed during driver initialization."3 An exporter that starts with the driver will publish a spike that is not work. Persistence mode removes the repeated driver init that keeps recreating this.
How to develop with it: a minimal real exporter¶
This section builds the exporter the Scope line promises: device enumeration, batched field reads with per-field status, the Xid event registration/wait path wired into a background watcher thread, a real HTTP /metrics endpoint serving Prometheus text-format output, and a scrape loop with SIGTERM/SIGINT-driven clean shutdown, all against the real nvidia-ml-py==13.610.43 API. The full module (nvml_exporter.py) is reproduced below exactly as written and executed; nothing in it is a mock of NVML. An earlier revision of this page defined watch_events and render_prometheus but never actually called the former from the main loop or served the latter over HTTP, and its __main__ block called only nvml_connect() before exiting, with no scrape loop and no signal handling; all four gaps are fixed below, not just documented as future work.
"""Minimal NVML exporter: init/shutdown, device enumeration, batched field
reads with per-field status, Xid event registration/wait wired into a
background watcher thread per device, a real HTTP /metrics endpoint in
Prometheus text-format output, and SIGTERM/SIGINT-driven clean shutdown.
Built against nvidia-ml-py==13.610.43 (import name: pynvml), the official
NVIDIA Python NVML binding. Every function in this file calls the real
library; nothing here is a mock of NVML's own behaviour. No third-party
import beyond pynvml: the HTTP server is Python's own http.server, and
shutdown is Python's own signal module.
"""
from __future__ import annotations
import signal
import threading
from dataclasses import dataclass, field
from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
import pynvml
# Field IDs read every scrape cycle. Chosen to cover a plain counter
# (ECC), a rate/instant reading (power), a temperature, and a link-health
# counter (PCIe replay) so per-field failure paths differ realistically.
FIELD_IDS: tuple[int, ...] = (
pynvml.NVML_FI_DEV_POWER_INSTANT,
pynvml.NVML_FI_DEV_MEMORY_TEMP,
pynvml.NVML_FI_DEV_ECC_SBE_VOL_TOTAL,
pynvml.NVML_FI_DEV_ECC_DBE_VOL_TOTAL,
pynvml.NVML_FI_DEV_PCIE_REPLAY_COUNTER,
)
FIELD_NAMES: dict[int, str] = {
pynvml.NVML_FI_DEV_POWER_INSTANT: "power_instant_mw",
pynvml.NVML_FI_DEV_MEMORY_TEMP: "memory_temp_celsius",
pynvml.NVML_FI_DEV_ECC_SBE_VOL_TOTAL: "ecc_sbe_volatile_total",
pynvml.NVML_FI_DEV_ECC_DBE_VOL_TOTAL: "ecc_dbe_volatile_total",
pynvml.NVML_FI_DEV_PCIE_REPLAY_COUNTER: "pcie_replay_total",
}
# nvmlFieldValue_t.valueType selects which member of the c_nvmlValue_t union
# holds the actual reading (nvml.h's nvmlValueType_t / pynvml's c_nvmlValue_t).
VALUE_ATTR: dict[int, str] = {
pynvml.NVML_VALUE_TYPE_DOUBLE: "dVal",
pynvml.NVML_VALUE_TYPE_UNSIGNED_INT: "uiVal",
pynvml.NVML_VALUE_TYPE_UNSIGNED_LONG: "ulVal",
pynvml.NVML_VALUE_TYPE_UNSIGNED_LONG_LONG: "ullVal",
pynvml.NVML_VALUE_TYPE_SIGNED_LONG_LONG: "sllVal",
pynvml.NVML_VALUE_TYPE_SIGNED_INT: "siVal",
pynvml.NVML_VALUE_TYPE_UNSIGNED_SHORT: "usVal",
}
@dataclass
class FieldReading:
name: str
ok: bool
value: float | int | None
error: str | None
latency_usec: int
@dataclass
class ExporterState:
"""Shared, lock-protected state between the scrape loop, the Xid
watcher threads, and the HTTP handler thread."""
lock: threading.Lock = field(default_factory=threading.Lock)
latest_text: str = "# no scrape has completed yet\n"
xid_counts: dict[int, int] = field(default_factory=dict)
field_read_error_counts: dict[int, int] = field(default_factory=dict)
def record_xid(self, device_index: int) -> None:
with self.lock:
self.xid_counts[device_index] = self.xid_counts.get(device_index, 0) + 1
def xid_count(self, device_index: int) -> int:
with self.lock:
return self.xid_counts.get(device_index, 0)
def record_field_read_errors(self, device_index: int, failures: int) -> int:
"""Add this scrape's failed fields once and return the process-lifetime
total. HTTP rendering never calls this method, so repeated reads of
/metrics cannot inflate the counter."""
if failures < 0:
raise ValueError("failures must be non-negative")
with self.lock:
total = self.field_read_error_counts.get(device_index, 0) + failures
self.field_read_error_counts[device_index] = total
return total
def field_read_error_count(self, device_index: int) -> int:
with self.lock:
return self.field_read_error_counts.get(device_index, 0)
def publish(self, text: str) -> None:
with self.lock:
self.latest_text = text
def read(self) -> str:
with self.lock:
return self.latest_text
class TerminalNVMLError(RuntimeError):
"""No usable NVML/driver on this host. Do not retry; this is a
configuration or environment defect, not a transient GPU condition."""
def nvml_connect() -> None:
"""Call once at process start (or after a full reconnect). Separates a
dead environment from a transient one so the caller's retry policy can
differ: a missing library or unloaded driver will not fix itself on the
next poll, so callers should exit non-zero / alert distinctly from a
per-field NOT_SUPPORTED or a Xid event."""
try:
pynvml.nvmlInit()
except pynvml.NVMLError_LibraryNotFound as exc:
raise TerminalNVMLError(
f"libnvidia-ml.so.1 not found ({exc}): no NVIDIA driver package "
"is installed on this host."
) from exc
except pynvml.NVMLError_DriverNotLoaded as exc:
raise TerminalNVMLError(
f"NVML library present but the driver is not loaded ({exc}): "
"the nvidia kernel module is not bound. This is the state a "
"long-running exporter must detect across a driver reload."
) from exc
except pynvml.NVMLError_NoPermission as exc:
raise TerminalNVMLError(f"NVML init denied by the OS ({exc})") from exc
def nvml_disconnect() -> None:
"""Best-effort shutdown; never let a teardown error mask the reason we
are disconnecting in the first place."""
try:
pynvml.nvmlShutdown()
except pynvml.NVMLError:
pass
def enumerate_devices() -> list:
"""nvmlDeviceGetCount can legitimately return 0 on a host with a live
driver and zero attached GPUs: NVML_SUCCESS, empty list. That is a
distinct, non-terminal condition from nvml_connect() failing outright,
and callers must not conflate the two."""
count = pynvml.nvmlDeviceGetCount()
return [pynvml.nvmlDeviceGetHandleByIndex(i) for i in range(count)]
def read_fields(handle) -> list[FieldReading]:
"""One nvmlDeviceGetFieldValues call for every configured field. The
top-level NVML_SUCCESS only means 'some values were populated' per
NVIDIA's own doc comment on the function; every element's own
.nvmlReturn must be checked before its .value is read, or an
unsupported field's undefined memory gets published as a metric."""
raw = pynvml.nvmlDeviceGetFieldValues(handle, list(FIELD_IDS))
readings: list[FieldReading] = []
for v in raw:
name = FIELD_NAMES[v.fieldId]
if v.nvmlReturn != pynvml.NVML_SUCCESS:
readings.append(FieldReading(
name=name, ok=False, value=None,
error=str(pynvml.NVMLError(v.nvmlReturn)),
latency_usec=v.latencyUsec,
))
continue
attr = VALUE_ATTR[v.valueType]
readings.append(FieldReading(
name=name, ok=True, value=getattr(v.value, attr), error=None,
latency_usec=v.latencyUsec,
))
return readings
def render_prometheus(
device_index: int,
readings: list[FieldReading],
field_error_total: int,
xid_total: int = 0,
) -> str:
"""Prometheus text exposition format. A failed field is dropped from the
gauge output (never zero-filled). field_error_total is the monotonic count
accumulated once per scrape by ExporterState, not the number of failures in
only the latest scrape; keeping mutation outside this renderer prevents an
HTTP GET from changing the counter. xid_total comes from the background
watcher thread, not from this scrape cycle: Xid events arrive asynchronously
between scrapes, not as a field read."""
lines: list[str] = []
for r in readings:
if r.ok:
lines.append(f'nvml_{r.name}{{gpu="{device_index}"}} {r.value}')
lines.append(f'nvml_field_read_errors_total{{gpu="{device_index}"}} {field_error_total}')
lines.append(f'nvml_xid_events_total{{gpu="{device_index}"}} {xid_total}')
return "\n".join(lines) + "\n"
def watch_events(handle, timeout_ms: int = 1000):
"""One poll of the Xid event path. nvmlEventSetWait's Python binding
raises NVMLError_Timeout on timeout (see pynvml.py's own comment,
"raises NVML_ERROR_TIMEOUT exception on timeout"); nvml.h documents this
as the function returning once the wait interval elapses with nothing
to report. Treat it as the steady state, not a failure: return None."""
event_set = pynvml.nvmlEventSetCreate()
try:
pynvml.nvmlDeviceRegisterEvents(
handle, pynvml.nvmlEventTypeXidCriticalError, event_set)
try:
return pynvml.nvmlEventSetWait(event_set, timeout_ms)
except pynvml.NVMLError_Timeout:
return None
finally:
pynvml.nvmlEventSetFree(event_set)
def xid_watcher_loop(
device_index: int,
handle,
state: ExporterState,
stop_event: threading.Event,
gpu_lost_event: threading.Event,
) -> None:
"""Runs in its own thread, one per device: nvmlEventSetWait blocks, so
this cannot share a thread with the scrape loop's periodic sleep. Each
non-None watch_events() return is a real Xid event; record it and keep
watching. NVMLError_GpuIsLost raised inside this thread (nvml.h documents
nvmlDeviceRegisterEvents itself as able to raise it) cannot reach
run_forever() by propagating: a background thread's target function has
no caller on another thread's call stack, so an uncaught exception here
only kills this thread silently (Python's default threading excepthook
logs a traceback and nothing else happens). gpu_lost_event is the
explicit signal that closes that gap: this thread sets it and returns,
and run_forever's main loop checks it every iteration to notice the
loss, disconnect, and force a reconnect. An earlier revision of this
function had no try/except here at all, so the error simply killed the
thread with no signal set: run_forever kept polling stale handles,
unaware the watcher was gone."""
while not stop_event.is_set():
try:
if watch_events(handle, timeout_ms=1000) is not None:
state.record_xid(device_index)
except pynvml.NVMLError_GpuIsLost:
gpu_lost_event.set()
return
class MetricsHTTPHandler(BaseHTTPRequestHandler):
"""Serves the most recently published Prometheus text on GET /metrics.
Reads exporter_state via the class attribute set by serve_metrics();
never touches NVML itself, only the lock-protected string the scrape
loop last published."""
exporter_state: ExporterState
def do_GET(self) -> None:
if self.path != "/metrics":
self.send_response(404)
self.end_headers()
return
body = self.exporter_state.read().encode("utf-8")
self.send_response(200)
self.send_header("Content-Type", "text/plain; version=0.0.4")
self.send_header("Content-Length", str(len(body)))
self.end_headers()
self.wfile.write(body)
def log_message(self, format: str, *args) -> None: # noqa: A002
pass # suppress BaseHTTPRequestHandler's default stderr access log
def serve_metrics(state: ExporterState, port: int) -> ThreadingHTTPServer:
"""Starts the real HTTP server in a background thread and returns it
immediately so the caller can run the scrape loop on the main thread
and call .shutdown() on this object during teardown."""
handler_cls = type("BoundMetricsHandler", (MetricsHTTPHandler,), {"exporter_state": state})
httpd = ThreadingHTTPServer(("0.0.0.0", port), handler_cls)
thread = threading.Thread(target=httpd.serve_forever, daemon=True)
thread.start()
return httpd
def scrape_once(handles: list, state: ExporterState) -> str:
chunks: list[str] = []
for i, handle in enumerate(handles):
readings = read_fields(handle)
failed_fields = sum(1 for reading in readings if not reading.ok)
field_error_total = state.record_field_read_errors(i, failed_fields)
chunks.append(render_prometheus(
i, readings, field_error_total=field_error_total, xid_total=state.xid_count(i)))
return "\n".join(chunks)
def start_xid_watchers(
handles: list, state: ExporterState
) -> tuple[list[threading.Thread], threading.Event, threading.Event]:
"""gpu_lost_event is shared by every watcher thread in this handle set:
any one device reporting GPU_IS_LOST forces the same full reconnect
run_forever already does for a scrape-path GPU_IS_LOST, since a lost bus
is not expected to be a single-device-only event and the exporter has no
per-device reconnect path today."""
stop_event = threading.Event()
gpu_lost_event = threading.Event()
threads = [
threading.Thread(
target=xid_watcher_loop, args=(i, h, state, stop_event, gpu_lost_event), daemon=True
)
for i, h in enumerate(handles)
]
for t in threads:
t.start()
return threads, stop_event, gpu_lost_event
def run_forever(
state: ExporterState,
port: int = 9400,
poll_interval_s: float = 15.0,
max_iterations: int | None = None,
shutdown_event: threading.Event | None = None,
) -> None:
"""The exporter's main loop: publishes to state.latest_text every
poll_interval_s so the real HTTP server above always has something to
serve, and runs one Xid watcher thread per device alongside it. On
NVMLError_GpuIsLost (documented as firing when a GPU falls off the bus,
and the error nvmlDeviceRegisterEvents itself can raise per nvml.h),
stop the watcher threads, drop the handle set, and force a full
nvml_connect()/enumerate_devices() cycle rather than continuing to poll
stale handles. A driver reload manifests through this same path.
GPU_IS_LOST reaches this loop two different ways, and both must drive
the same teardown: it can be raised directly by a scrape-path NVML call
on this thread (caught below), or it can be detected first by a
background xid_watcher_loop thread, which has no call stack back into
this thread and so can only set watcher_gpu_lost -- checked once per
iteration, right after publishing. An earlier revision of this function
only handled the caught-exception case: a watcher-detected loss left the
watcher thread dead and this loop none the wiser, still polling the
stale handle set. `gpu_lost` below unifies both signals into one
teardown so neither path is silently missed.
shutdown_event, when supplied by main() via install_signal_handlers(),
turns the poll wait into shutdown_event.wait(poll_interval_s) instead of
time.sleep(poll_interval_s): a SIGTERM/SIGINT sets the event and the wait
returns immediately instead of blocking for the rest of the interval, so
the loop exits and the finally block below tears down the watcher
threads and the HTTP server without waiting out a stale poll period."""
shutdown_event = shutdown_event if shutdown_event is not None else threading.Event()
handles: list | None = None
watcher_stop: threading.Event | None = None
watcher_gpu_lost: threading.Event | None = None
httpd = serve_metrics(state, port)
iterations = 0
try:
while (max_iterations is None or iterations < max_iterations) and not shutdown_event.is_set():
gpu_lost = False
try:
if handles is None:
nvml_connect()
handles = enumerate_devices()
_, watcher_stop, watcher_gpu_lost = start_xid_watchers(handles, state)
state.publish(scrape_once(handles, state))
gpu_lost = watcher_gpu_lost is not None and watcher_gpu_lost.is_set()
except pynvml.NVMLError_GpuIsLost:
gpu_lost = True
if gpu_lost:
if watcher_stop is not None:
watcher_stop.set()
nvml_disconnect()
handles = None
watcher_stop = None
watcher_gpu_lost = None
iterations += 1
more_iterations = max_iterations is None or iterations < max_iterations
if more_iterations and not shutdown_event.is_set():
shutdown_event.wait(poll_interval_s)
finally:
if watcher_stop is not None:
watcher_stop.set()
httpd.shutdown()
def install_signal_handlers(shutdown_event: threading.Event) -> None:
"""SIGTERM is what a process manager (systemd, Kubernetes) sends to ask
for a clean stop; SIGINT is Ctrl-C. Neither is handled in a way that
guarantees run_forever's finally block runs promptly without this:
SIGTERM's default action kills the process outright, bypassing Python
entirely, and even SIGINT's default KeyboardInterrupt only unwinds after
the interpreter next checks for it. Setting shutdown_event makes both
signals interrupt an in-progress poll wait immediately and take the same
clean-exit path through run_forever's finally block."""
def _request_shutdown(signum: int, frame) -> None:
shutdown_event.set()
signal.signal(signal.SIGTERM, _request_shutdown)
signal.signal(signal.SIGINT, _request_shutdown)
def main() -> None:
"""The real entry point: installs SIGTERM/SIGINT handlers before
starting the loop, then run_forever() owns the connect/reconnect
lifecycle (its first iteration connects, same as its GpuIsLost recovery
path), so main() only needs to guarantee a final nvmlShutdown() on the
way out, whether run_forever returns normally, is signalled to stop, or
a TerminalNVMLError escapes its first connect attempt. An earlier
revision of this block called only nvml_connect() and exited: no scrape
loop, no HTTP server, no shutdown, and no signal handling."""
state = ExporterState()
shutdown_event = threading.Event()
install_signal_handlers(shutdown_event)
try:
run_forever(state, shutdown_event=shutdown_event)
finally:
nvml_disconnect()
if __name__ == "__main__":
main()
Run directly on this sandbox, main() -> run_forever() -> nvml_connect() fails exactly as shown above (the DriverNotLoaded variant, the steady state once the background package install completed) and exits non-zero, which is the correct behavior for a process manager to restart-and-alert on rather than silently loop. This is the real, complete traceback, not truncated for the page: it now runs through main() and run_forever() because __main__ calls main() instead of stopping after a single nvml_connect() call:
$ python3 nvml_exporter.py
Traceback (most recent call last):
File "nvml_exporter.py", line 115, in nvml_connect
pynvml.nvmlInit()
File "pynvml.py", line 2991, in nvmlInit
nvmlInitWithFlags(0)
File "pynvml.py", line 2981, in nvmlInitWithFlags
_nvmlCheckReturn(ret)
File "pynvml.py", line 1098, in _nvmlCheckReturn
raise NVMLError(ret)
pynvml.NVMLError_DriverNotLoaded: Driver Not Loaded
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "nvml_exporter.py", line 413, in <module>
main()
File "nvml_exporter.py", line 407, in main
run_forever(state, shutdown_event=shutdown_event)
File "nvml_exporter.py", line 355, in run_forever
nvml_connect()
File "nvml_exporter.py", line 122, in nvml_connect
raise TerminalNVMLError(
TerminalNVMLError: NVML library present but the driver is not loaded (Driver Not Loaded): the nvidia kernel module is not bound. This is the state a long-running exporter must detect across a driver reload.
$ echo "exit=$?"
exit=1
Testing what a live device cannot be tested against here¶
With no GPU in this sandbox, read_fields, watch_events, and the recovery path in run_forever cannot be driven by a real device. Each is instead tested against the real ctypes structures and real exception classes nvidia-ml-py defines, with exactly one library call monkeypatched per test (labelled at the call site) so the test exercises this page's own logic, not a fabrication of NVML's behavior. Three tests drive the Xid watcher, real stdlib HTTP endpoint, and real SIGTERM path. Two more close the background-thread recovery gap an adversarial rerun found: one confirms xid_watcher_loop turns a real NVMLError_GpuIsLost into the shared signal instead of dying silently, and the other confirms run_forever disconnects and reconnects after that signal. The field-error regression test drives three consecutive scrapes and then calls the renderer again, proving the _total value progresses 1 -> 2 -> 2 and that HTTP rendering is side-effect free:
"""Executed, assert-validated tests for nvml_exporter.py."""
from __future__ import annotations
import os
import signal
import threading
import time
import urllib.request
from unittest import mock
import pynvml
import nvml_exporter as exp
def test_real_init_fails_with_no_driver() -> None:
"""No mocking: this sandbox genuinely has no bound NVIDIA kernel driver.
nvml_connect() must convert the real NVMLError into TerminalNVMLError,
not let a bare NVMLError escape to the caller."""
try:
exp.nvml_connect()
except exp.TerminalNVMLError as e:
print(f"real nvml_connect() failed as expected, converted: {e}")
return
raise AssertionError("expected TerminalNVMLError: a GPU driver appears to be present")
def test_terminal_error_classification_is_exhaustive() -> None:
"""Each of the three real NVMLError subclasses nvml_connect() special-
cases must route to TerminalNVMLError. Constructed directly (these are
the real pynvml exception classes, not stand-ins) and run through the
same except-block nvml_connect() uses, via a thin call-through."""
for real_exc in (pynvml.NVMLError_LibraryNotFound(),
pynvml.NVMLError_DriverNotLoaded(),
pynvml.NVMLError_NoPermission()):
with mock.patch("pynvml.nvmlInit", side_effect=real_exc):
try:
exp.nvml_connect()
except exp.TerminalNVMLError:
continue
raise AssertionError(f"{type(real_exc).__name__} did not raise TerminalNVMLError")
print("all 3 real terminal NVMLError subclasses route to TerminalNVMLError")
def test_read_fields_separates_per_field_success_and_failure() -> None:
"""Populate the REAL c_nvmlFieldValue_t ctypes structure (from pynvml,
not reimplemented) by hand, since no device exists to populate it via a
live call. One field succeeds (NVML_SUCCESS, unsigned-int value), one
field fails (NVML_ERROR_NOT_SUPPORTED). read_fields must not raise, must
not fabricate a value for the failed field, and must report both."""
values = (pynvml.c_nvmlFieldValue_t * 2)()
values[0].fieldId = pynvml.NVML_FI_DEV_POWER_INSTANT
values[0].nvmlReturn = pynvml.NVML_SUCCESS
values[0].valueType = pynvml.NVML_VALUE_TYPE_UNSIGNED_INT
values[0].value.uiVal = 42000
values[0].latencyUsec = 55
values[1].fieldId = pynvml.NVML_FI_DEV_MEMORY_TEMP
values[1].nvmlReturn = pynvml.NVML_ERROR_NOT_SUPPORTED
values[1].latencyUsec = 12
with mock.patch("pynvml.nvmlDeviceGetFieldValues", return_value=values):
readings = exp.read_fields(handle=None)
assert len(readings) == 2
power, temp = readings
assert power.ok is True and power.value == 42000 and power.error is None
assert temp.ok is False and temp.value is None
assert temp.error == str(pynvml.NVMLError(pynvml.NVML_ERROR_NOT_SUPPORTED))
print(f"power reading: ok={power.ok} value={power.value}")
print(f"temp reading: ok={temp.ok} error={temp.error!r}")
def test_prometheus_output_omits_failed_fields_but_counts_them() -> None:
readings = [
exp.FieldReading("power_instant_mw", True, 42000, None, 55),
exp.FieldReading("memory_temp_celsius", False, None, "Not Supported", 12),
]
text = exp.render_prometheus(0, readings, field_error_total=1, xid_total=3)
assert 'nvml_power_instant_mw{gpu="0"} 42000' in text
assert "memory_temp_celsius" not in text # failed field never zero-filled
assert 'nvml_field_read_errors_total{gpu="0"} 1' in text
assert 'nvml_xid_events_total{gpu="0"} 3' in text
print("Prometheus output:")
print(text, end="")
def test_field_read_error_total_is_monotonic_across_scrapes() -> None:
"""A `_total` counter must accumulate failed fields across scrapes, must
not fall when a later scrape is clean, and must not change merely because
Prometheus rendering runs again."""
failed = [
exp.FieldReading("memory_temp_celsius", False, None, "Not Supported", 12),
]
healthy = [
exp.FieldReading("power_instant_mw", True, 42000, None, 55),
]
state = exp.ExporterState()
with mock.patch.object(exp, "read_fields", side_effect=[failed, failed, healthy]):
first = exp.scrape_once(["fake-handle"], state)
second = exp.scrape_once(["fake-handle"], state)
third = exp.scrape_once(["fake-handle"], state)
assert 'nvml_field_read_errors_total{gpu="0"} 1' in first
assert 'nvml_field_read_errors_total{gpu="0"} 2' in second
assert 'nvml_field_read_errors_total{gpu="0"} 2' in third
assert state.field_read_error_count(0) == 2
exp.render_prometheus(0, failed, field_error_total=2, xid_total=0)
assert state.field_read_error_count(0) == 2, "rendering must be side-effect free"
print("field read error counter across scrapes: 1 -> 2 -> 2; render is side-effect free")
def test_watch_events_timeout_is_not_an_error() -> None:
"""nvmlEventSetWait's Python binding raises NVMLError_Timeout on
timeout (pynvml.py's own comment says so). Monkeypatch only the wait
call, with the real exception class, and confirm watch_events()
returns None rather than propagating."""
with mock.patch("pynvml.nvmlEventSetCreate", return_value=object()), \
mock.patch("pynvml.nvmlDeviceRegisterEvents"), \
mock.patch("pynvml.nvmlEventSetWait", side_effect=pynvml.NVMLError_Timeout()), \
mock.patch("pynvml.nvmlEventSetFree"):
result = exp.watch_events(handle=None, timeout_ms=50)
assert result is None
print("watch_events() returned None on NVMLError_Timeout, as documented steady state")
def test_register_events_gpu_is_lost_propagates_not_swallowed() -> None:
"""nvml.h documents nvmlDeviceRegisterEvents itself as able to raise
NVML_ERROR_GPU_IS_LOST. watch_events must not catch this (only Timeout
is benign); it has to reach the caller's reconnect logic."""
with mock.patch("pynvml.nvmlEventSetCreate", return_value=object()), \
mock.patch("pynvml.nvmlDeviceRegisterEvents", side_effect=pynvml.NVMLError_GpuIsLost()), \
mock.patch("pynvml.nvmlEventSetFree") as free_mock:
try:
exp.watch_events(handle=None, timeout_ms=50)
except pynvml.NVMLError_GpuIsLost:
assert free_mock.called # event set must still be freed via `finally`
print("NVMLError_GpuIsLost from RegisterEvents propagated; event set was freed")
return
raise AssertionError("expected NVMLError_GpuIsLost to propagate")
def test_xid_watcher_thread_records_a_real_event_then_stops() -> None:
"""Drives the background xid_watcher_loop thread this page's earlier
revision never wired into run_forever at all. Monkeypatches only
nvmlEventSetWait: first call returns a real Xid code (999, "unknown Xid
error" per nvml.h), second call raises the real NVMLError_Timeout so the
loop's next iteration finds stop_event set and exits. Verifies the
thread actually calls watch_events(), actually increments
ExporterState.xid_counts, and actually terminates on stop_event."""
state = exp.ExporterState()
stop_event = threading.Event()
gpu_lost_event = threading.Event()
call_count = {"n": 0}
def fake_wait(event_set, timeout_ms):
call_count["n"] += 1
if call_count["n"] == 1:
return 999
stop_event.set()
raise pynvml.NVMLError_Timeout()
with mock.patch("pynvml.nvmlEventSetCreate", return_value=object()), \
mock.patch("pynvml.nvmlDeviceRegisterEvents"), \
mock.patch("pynvml.nvmlEventSetWait", side_effect=fake_wait), \
mock.patch("pynvml.nvmlEventSetFree"):
thread = threading.Thread(
target=exp.xid_watcher_loop, args=(0, None, state, stop_event, gpu_lost_event)
)
thread.start()
thread.join(timeout=5)
assert not thread.is_alive(), "watcher thread did not stop on stop_event"
assert state.xid_count(0) == 1
assert not gpu_lost_event.is_set(), "gpu_lost_event must stay clear on a normal timeout/Xid path"
print(f"xid_watcher_loop recorded {state.xid_count(0)} real Xid event(s) then stopped cleanly")
def test_watcher_gpu_is_lost_sets_shared_event_not_swallowed() -> None:
"""The defect this page fixes: NVMLError_GpuIsLost raised inside
xid_watcher_loop (via nvmlDeviceRegisterEvents, a documented raise point)
has no Python call stack back to run_forever's thread, so it cannot
propagate there. Confirms the loop itself catches it, sets
gpu_lost_event, and returns -- the explicit signal run_forever now polls
-- instead of the exception being silently swallowed or killing the
thread with nothing to show for it. Against the page's prior revision,
this test fails: the exception escapes the thread target uncaught,
gpu_lost_event is never set, and only a default-excepthook traceback on
stderr shows the thread died."""
state = exp.ExporterState()
stop_event = threading.Event()
gpu_lost_event = threading.Event()
with mock.patch("pynvml.nvmlEventSetCreate", return_value=object()), \
mock.patch("pynvml.nvmlDeviceRegisterEvents", side_effect=pynvml.NVMLError_GpuIsLost()), \
mock.patch("pynvml.nvmlEventSetFree"):
thread = threading.Thread(
target=exp.xid_watcher_loop, args=(0, None, state, stop_event, gpu_lost_event)
)
thread.start()
thread.join(timeout=5)
assert not thread.is_alive(), "watcher thread must exit on GPU_IS_LOST, not hang"
assert gpu_lost_event.is_set(), "gpu_lost_event must be set so run_forever can observe the loss"
print("xid_watcher_loop caught real NVMLError_GpuIsLost, set gpu_lost_event, and returned")
def test_http_metrics_endpoint_serves_latest_published_text() -> None:
"""Drives the real stdlib HTTP server this page's earlier revision
never included at all: starts serve_metrics() on an ephemeral port,
publishes text through the same ExporterState.publish() the scrape loop
uses, and issues a real HTTP GET against it with urllib, not a
simulated request object."""
state = exp.ExporterState()
state.publish('nvml_power_instant_mw{gpu="0"} 42000\n')
httpd = exp.serve_metrics(state, port=0) # port 0: OS picks a free port
port = httpd.server_address[1]
try:
time.sleep(0.1) # let the server thread reach serve_forever()
with urllib.request.urlopen(f"http://127.0.0.1:{port}/metrics", timeout=5) as resp:
status = resp.status
body = resp.read().decode("utf-8")
try:
urllib.request.urlopen(f"http://127.0.0.1:{port}/not-metrics", timeout=5)
except urllib.error.HTTPError as e:
not_found_status = e.code
finally:
httpd.shutdown()
assert status == 200
assert body == 'nvml_power_instant_mw{gpu="0"} 42000\n'
assert not_found_status == 404
print(f"GET /metrics -> {status}, body={body!r}")
print(f"GET /not-metrics -> {not_found_status} (real 404, not simulated)")
def test_run_forever_wires_http_server_and_xid_watchers_and_recovers() -> None:
"""Driver-restart-recovery simulation, explicitly marked: mock
scrape_once to raise the real NVMLError_GpuIsLost on the first
iteration, and mock nvml_connect/enumerate_devices/nvml_disconnect so
the test exercises ONLY run_forever's own state machine (drop handles,
disconnect, force reconnect, restart watchers), not real NVML. Also
confirms run_forever calls serve_metrics() and start_xid_watchers() at
all -- an earlier revision's __main__ never called run_forever, so
neither the HTTP server nor the watcher threads were ever started."""
calls = {"connect": 0, "disconnect": 0, "scrape": 0, "serve": 0, "watchers": 0, "shutdown": 0}
def fake_connect() -> None:
calls["connect"] += 1
def fake_enumerate() -> list[str]:
return ["fake-handle"]
def fake_disconnect() -> None:
calls["disconnect"] += 1
def fake_scrape(handles: list, state: exp.ExporterState) -> str:
calls["scrape"] += 1
if calls["scrape"] == 1:
raise pynvml.NVMLError_GpuIsLost()
return "ok"
class FakeHTTPD:
def shutdown(self) -> None:
calls["shutdown"] += 1
def fake_serve(state: exp.ExporterState, port: int):
calls["serve"] += 1
return FakeHTTPD()
def fake_start_watchers(handles: list, state: exp.ExporterState):
calls["watchers"] += 1
return [], threading.Event(), threading.Event()
state = exp.ExporterState()
with mock.patch.object(exp, "nvml_connect", fake_connect), \
mock.patch.object(exp, "enumerate_devices", fake_enumerate), \
mock.patch.object(exp, "nvml_disconnect", fake_disconnect), \
mock.patch.object(exp, "scrape_once", fake_scrape), \
mock.patch.object(exp, "serve_metrics", fake_serve), \
mock.patch.object(exp, "start_xid_watchers", fake_start_watchers):
exp.run_forever(state, poll_interval_s=0, max_iterations=2)
assert calls["connect"] == 2, "must reconnect after GPU_IS_LOST"
assert calls["disconnect"] == 1, "must disconnect the stale handle set before reconnecting"
assert calls["scrape"] == 2
assert calls["serve"] == 1, "run_forever must start the real HTTP server exactly once"
assert calls["watchers"] == 2, "must restart xid watchers on every (re)connect, including after GPU_IS_LOST"
assert calls["shutdown"] == 1, "must shut down the HTTP server on exit"
print(f"run_forever recovery: connect={calls['connect']} disconnect={calls['disconnect']} "
f"scrape={calls['scrape']} serve={calls['serve']} watchers={calls['watchers']} "
f"shutdown={calls['shutdown']}")
def test_run_forever_recovers_from_watcher_detected_gpu_is_lost() -> None:
"""The adversarial scenario an earlier revision failed: NVMLError_GpuIsLost
raised from inside the background xid_watcher_loop thread, not from the
scrape path. Unlike the test above, start_xid_watchers here is the REAL
function, driving a REAL background thread running the REAL
xid_watcher_loop; only the pynvml-level event calls are mocked (real
exception classes), and only nvml_connect/enumerate_devices/
nvml_disconnect/scrape_once/serve_metrics are replaced with counters, so
this isolates the watcher-driven trigger from the scrape-driven one the
test above already covers. An adversarial rerun against the prior
revision of run_forever observed exactly the failure this test guards
against: one connect, no disconnect, scrapes continuing regardless, and
a watcher thread that died silently."""
calls = {"connect": 0, "disconnect": 0, "scrape": 0}
register_calls = {"n": 0}
def fake_connect() -> None:
calls["connect"] += 1
def fake_enumerate() -> list[str]:
return ["fake-handle"]
def fake_disconnect() -> None:
calls["disconnect"] += 1
def fake_scrape(handles: list, state: exp.ExporterState) -> str:
calls["scrape"] += 1
return "ok"
class FakeHTTPD:
def shutdown(self) -> None:
pass
def fake_serve(state: exp.ExporterState, port: int):
return FakeHTTPD()
def fake_register_events(handle, event_types, event_set):
register_calls["n"] += 1
if register_calls["n"] == 1:
raise pynvml.NVMLError_GpuIsLost()
def fake_wait(event_set, timeout_ms):
raise pynvml.NVMLError_Timeout()
state = exp.ExporterState()
baseline_threads = set(threading.enumerate())
with mock.patch.object(exp, "nvml_connect", fake_connect), \
mock.patch.object(exp, "enumerate_devices", fake_enumerate), \
mock.patch.object(exp, "nvml_disconnect", fake_disconnect), \
mock.patch.object(exp, "scrape_once", fake_scrape), \
mock.patch.object(exp, "serve_metrics", fake_serve), \
mock.patch("pynvml.nvmlEventSetCreate", return_value=object()), \
mock.patch("pynvml.nvmlDeviceRegisterEvents", side_effect=fake_register_events), \
mock.patch("pynvml.nvmlEventSetWait", side_effect=fake_wait), \
mock.patch("pynvml.nvmlEventSetFree"):
exp.run_forever(state, poll_interval_s=0.05, max_iterations=6)
# run_forever's own finally already calls watcher_stop.set(), but does
# not join the thread; join it here, while the pynvml patches above
# are still active, so the real background watcher thread's next loop
# check cannot race against those patches being reverted the moment
# this `with` block exits.
for leftover in set(threading.enumerate()) - baseline_threads:
leftover.join(timeout=2)
assert calls["connect"] == 2, "watcher-detected GPU_IS_LOST must still trigger a reconnect"
assert calls["disconnect"] == 1, "must disconnect the stale handle set exactly once"
assert calls["scrape"] == 6, "scrapes must keep running across the reconnect, not stall"
assert register_calls["n"] >= 2, "the restarted watcher must re-register events after reconnect"
watcher_re_registered = register_calls["n"] >= 2
assert watcher_re_registered
print(f"watcher-triggered recovery: connect={calls['connect']} disconnect={calls['disconnect']} "
f"scrape={calls['scrape']} watcher_re_registered={watcher_re_registered}")
def test_sigterm_triggers_clean_shutdown_of_run_forever() -> None:
"""Real OS signal delivery, not a simulated call: install_signal_handlers()
registers real handlers via signal.signal(), run_forever() runs on a
background thread with its NVML calls mocked (same technique as the
recovery test above, so this exercises the signal path, not a live
device) behind a real HTTP server via the real serve_metrics(), and the
test sends a real SIGTERM to this process with os.kill(). poll_interval_s
is set to 30s, so a pass here only works if the signal actually
interrupted the wait; the original handlers are restored afterward so
this test does not leak signal state into the rest of the run. Elapsed time
is measured and checked against a bound but deliberately not printed; stdout
reports only the asserted, deterministic pass condition."""
calls = {"scrape": 0}
def fake_scrape(handles: list, state: exp.ExporterState) -> str:
calls["scrape"] += 1
return "ok"
def fake_start_watchers(handles: list, state: exp.ExporterState):
return [], threading.Event(), threading.Event()
shutdown_event = threading.Event()
original_term = signal.getsignal(signal.SIGTERM)
original_int = signal.getsignal(signal.SIGINT)
exp.install_signal_handlers(shutdown_event)
try:
with mock.patch.object(exp, "nvml_connect", lambda: None), \
mock.patch.object(exp, "enumerate_devices", lambda: ["fake-handle"]), \
mock.patch.object(exp, "nvml_disconnect", lambda: None), \
mock.patch.object(exp, "scrape_once", fake_scrape), \
mock.patch.object(exp, "start_xid_watchers", fake_start_watchers):
state = exp.ExporterState()
thread = threading.Thread(
target=exp.run_forever,
args=(state,),
kwargs={"port": 0, "poll_interval_s": 30.0, "shutdown_event": shutdown_event},
)
thread.start()
time.sleep(0.2) # let the loop reach its first shutdown_event.wait()
start = time.monotonic()
os.kill(os.getpid(), signal.SIGTERM)
thread.join(timeout=5)
elapsed = time.monotonic() - start
finally:
signal.signal(signal.SIGTERM, original_term)
signal.signal(signal.SIGINT, original_int)
assert not thread.is_alive(), "run_forever did not stop after a real SIGTERM"
interrupted_before_bound = elapsed < 5.0
assert interrupted_before_bound, (
f"SIGTERM took {elapsed:.1f}s against a 30s poll wait: not actually interrupted")
assert calls["scrape"] >= 1
print("SIGTERM interrupted 30-second poll before 5-second bound: "
f"{interrupted_before_bound}")
if __name__ == "__main__":
for t in (test_real_init_fails_with_no_driver,
test_terminal_error_classification_is_exhaustive,
test_read_fields_separates_per_field_success_and_failure,
test_prometheus_output_omits_failed_fields_but_counts_them,
test_field_read_error_total_is_monotonic_across_scrapes,
test_watch_events_timeout_is_not_an_error,
test_register_events_gpu_is_lost_propagates_not_swallowed,
test_xid_watcher_thread_records_a_real_event_then_stops,
test_watcher_gpu_is_lost_sets_shared_event_not_swallowed,
test_http_metrics_endpoint_serves_latest_published_text,
test_run_forever_wires_http_server_and_xid_watchers_and_recovers,
test_run_forever_recovers_from_watcher_detected_gpu_is_lost,
test_sigterm_triggers_clean_shutdown_of_run_forever):
print(f"--- {t.__name__} ---")
t()
print("\nall tests passed")
Executed output, in this sandbox, reproduced byte-for-byte on rerun. Tests may measure a live duration or execute a timing-dependent number of mocked calls internally, but stdout contains only the deterministic facts each test asserts, never those unstable raw values:
--- test_real_init_fails_with_no_driver ---
real nvml_connect() failed as expected, converted: NVML library present but the driver is not loaded (Driver Not Loaded): the nvidia kernel module is not bound. This is the state a long-running exporter must detect across a driver reload.
--- test_terminal_error_classification_is_exhaustive ---
all 3 real terminal NVMLError subclasses route to TerminalNVMLError
--- test_read_fields_separates_per_field_success_and_failure ---
power reading: ok=True value=42000
temp reading: ok=False error='Not Supported'
--- test_prometheus_output_omits_failed_fields_but_counts_them ---
Prometheus output:
nvml_power_instant_mw{gpu="0"} 42000
nvml_field_read_errors_total{gpu="0"} 1
nvml_xid_events_total{gpu="0"} 3
--- test_field_read_error_total_is_monotonic_across_scrapes ---
field read error counter across scrapes: 1 -> 2 -> 2; render is side-effect free
--- test_watch_events_timeout_is_not_an_error ---
watch_events() returned None on NVMLError_Timeout, as documented steady state
--- test_register_events_gpu_is_lost_propagates_not_swallowed ---
NVMLError_GpuIsLost from RegisterEvents propagated; event set was freed
--- test_xid_watcher_thread_records_a_real_event_then_stops ---
xid_watcher_loop recorded 1 real Xid event(s) then stopped cleanly
--- test_watcher_gpu_is_lost_sets_shared_event_not_swallowed ---
xid_watcher_loop caught real NVMLError_GpuIsLost, set gpu_lost_event, and returned
--- test_http_metrics_endpoint_serves_latest_published_text ---
GET /metrics -> 200, body='nvml_power_instant_mw{gpu="0"} 42000\n'
GET /not-metrics -> 404 (real 404, not simulated)
--- test_run_forever_wires_http_server_and_xid_watchers_and_recovers ---
run_forever recovery: connect=2 disconnect=1 scrape=2 serve=1 watchers=2 shutdown=1
--- test_run_forever_recovers_from_watcher_detected_gpu_is_lost ---
watcher-triggered recovery: connect=2 disconnect=1 scrape=6 watcher_re_registered=True
--- test_sigterm_triggers_clean_shutdown_of_run_forever ---
SIGTERM interrupted 30-second poll before 5-second bound: True
all tests passed
The watcher-recovery test asserts that registration happened at least twice, and the SIGTERM test asserts that a 30-second poll wait ended inside a 5-second bound. Their raw call count and elapsed time vary with thread scheduling, so the transcript reports the asserted booleans instead. This keeps the evidence honest and makes the full captured stdout reproducible byte-for-byte.
Ten things this validates that a bitmask decoder or a header reproduction cannot: (1) all three terminal init failures reach the same fail-fast path, not just the one this host happens to produce; (2) a batched field read with one supported and one unsupported field neither crashes nor fabricates a value for the unsupported one; (3) failed fields accumulate in a real monotonic _total counter once per scrape, a clean scrape does not decrement it, and rendering does not mutate it; (4) an event-wait timeout, the path an exporter hits on every single poll of an idle GPU, is treated as success, while a real GPU_IS_LOST from event registration is not swallowed; (5) the main loop actually drops its stale device handles and re-runs the connect/enumerate sequence before resuming, rather than continuing to poll a handle table from before the loss; (6) the Xid watcher actually runs as a background thread, actually records a real event, and actually stops on its stop_event, not just defined and unit-tested in isolation from the loop that was supposed to start it; (7) a real GPU_IS_LOST raised inside that background thread is caught there and signalled out via gpu_lost_event rather than killing the thread with nothing for run_forever to observe, which is exactly the gap an adversarial rerun found in an earlier revision (one connect, no disconnect, and a dead watcher); (8) run_forever itself, driving a real watcher thread rather than a faked one, actually notices that signal and disconnects and reconnects rather than polling stale handles forever; (9) the /metrics endpoint is a real, running ThreadingHTTPServer a real urllib client can GET, not a Prometheus-format string that only ever reaches print(); (10) a real SIGTERM delivered with os.kill() interrupts an in-progress 30-second poll wait inside the asserted bound and drives the same clean-exit path as a normal loop exit, rather than requiring a process manager to escalate to SIGKILL after a stuck shutdown.
Both _total counters are process-lifetime counters: they reset when the exporter process restarts and survive an in-process NVML reconnect. This minimal exporter keys state and labels by enumeration index, so a driver reload that changes device order can associate the retained history with a different physical GPU. A fleet exporter should key state and label series by nvmlDeviceGetUUID() instead; this example keeps the index-based shape small and states the limitation rather than implying stable identity.
Interpreting one field: the clock-event-reason bitmask¶
The other number operators reach for is "why is this GPU slow", answered by the clock event reason bitmask from nvmlDeviceGetCurrentClocksEventReasons. Three things about it are easy to get wrong, and all three are in NVIDIA's own header.
The rename is real, and it is partial. Six reasons plus the None/All sentinels were renamed from nvmlClocksThrottleReason* to nvmlClocksEventReason*, with the old spellings retained as deprecated aliases of the same value. The three hardware reasons were not renamed at all: nvmlClocksThrottleReasonHwSlowdown (0x8), nvmlClocksThrottleReasonHwThermalSlowdown (0x40) and nvmlClocksThrottleReasonHwPowerBrakeSlowdown (0x80) keep their original names as the current, primary spelling. There is no nvmlClocksEventReasonHwSlowdown.3 A blanket search-and-replace of the identifier across your codebase will not compile. The bit values never changed, so any decoder that works on values is unaffected.
Two separate events get conflated here, so be precise about which one bit you. The nvmlClocksEventReason* macros already exist in headers shipping NVML_API_VERSION 12; what arrived with NVML_API_VERSION 13 is the DEPRECATED(13.0) attribute on the old functions, which is when the compiler started complaining.5 So the macros changed in one release and the warnings appeared in a later one.
A nonzero mask is not a fault. GpuIdle (0x1) is a reason. An idle GPU reports a nonzero mask, and the naive if (mask) alert() pages you at 3am because nobody is using the machine.
HwSlowdown alone is ambiguous, by NVIDIA's own admission. The header lists its causes as temperature too high, external power brake asserted, and fast-trigger power protection, and then adds two further bullets: "May be also reported during PState or clock change" and, under it, "This behavior may be removed in a later release."3 So a single sample with HwSlowdown set can just be a clock transition. NVIDIA does not document that HwSlowdown is always co-asserted with the specific thermal or power-brake bits, so do not assume it: alert on the specific bits immediately, and on bare HwSlowdown only when it persists.
The decoder below is the arithmetic that follows from those three facts, with the bit values read out of nvml.h.
"""Decoding the NVML clock-event bitmask into an alert decision.
Bit values are read from NVIDIA's nvml.h (NVML_API_VERSION 13). Note the spelling:
the three hardware reasons kept the nvmlClocksThrottleReason* name; the other six
(plus the None/All sentinels) were renamed to nvmlClocksEventReason*. Values unchanged.
"""
import numpy as np
# --- nvml.h, NVML_API_VERSION 13 ------------------------------------------
NONE = 0x0000000000000000 # nvmlClocksEventReasonNone
GPU_IDLE = 0x0000000000000001 # nvmlClocksEventReasonGpuIdle
APPLICATIONS_CLOCKS_SETTING = 0x0000000000000002 # nvmlClocksEventReasonApplicationsClocksSetting
SW_POWER_CAP = 0x0000000000000004 # nvmlClocksEventReasonSwPowerCap
HW_SLOWDOWN = 0x0000000000000008 # nvmlClocksThrottleReasonHwSlowdown (NOT renamed)
SYNC_BOOST = 0x0000000000000010 # nvmlClocksEventReasonSyncBoost
SW_THERMAL_SLOWDOWN = 0x0000000000000020 # nvmlClocksEventReasonSwThermalSlowdown
HW_THERMAL_SLOWDOWN = 0x0000000000000040 # nvmlClocksThrottleReasonHwThermalSlowdown (NOT renamed)
HW_POWER_BRAKE_SLOWDOWN = 0x0000000000000080 # nvmlClocksThrottleReasonHwPowerBrakeSlowdown (NOT renamed)
DISPLAY_CLOCK_SETTING = 0x0000000000000100 # nvmlClocksEventReasonDisplayClockSetting
ALL = (GPU_IDLE | APPLICATIONS_CLOCKS_SETTING | SW_POWER_CAP | HW_SLOWDOWN | SYNC_BOOST
| SW_THERMAL_SLOWDOWN | HW_THERMAL_SLOWDOWN | HW_POWER_BRAKE_SLOWDOWN
| DISPLAY_CLOCK_SETTING) # nvmlClocksEventReasonAll
assert ALL == 0x1FF
# Severity, not a flat list. "Throttled" is not the same as "broken".
BENIGN = GPU_IDLE # no work to do
OPERATOR = APPLICATIONS_CLOCKS_SETTING | DISPLAY_CLOCK_SETTING | SYNC_BOOST # someone set this
DERATE = SW_POWER_CAP | SW_THERMAL_SLOWDOWN # firmware is managing us
FAULT = HW_THERMAL_SLOWDOWN | HW_POWER_BRAKE_SLOWDOWN # hardware protection fired
NAMES = {GPU_IDLE: "GpuIdle", APPLICATIONS_CLOCKS_SETTING: "ApplicationsClocksSetting",
SW_POWER_CAP: "SwPowerCap", HW_SLOWDOWN: "HwSlowdown", SYNC_BOOST: "SyncBoost",
SW_THERMAL_SLOWDOWN: "SwThermalSlowdown", HW_THERMAL_SLOWDOWN: "HwThermalSlowdown",
HW_POWER_BRAKE_SLOWDOWN: "HwPowerBrakeSlowdown",
DISPLAY_CLOCK_SETTING: "DisplayClockSetting"}
def decode(mask):
"""Bits set, plus any bit NVML added that this build does not know about."""
known = sorted(b for b in NAMES if mask & b)
return [NAMES[b] for b in known], mask & ~ALL # (names, unknown_bits)
def classify(mask):
"""Severity of a single sample. HwSlowdown alone is deliberately not a fault."""
_, unknown = decode(mask)
if unknown:
return "unknown" # a future NVML bit: surface it, never drop it
if mask & FAULT:
return "fault"
if mask & DERATE:
return "derate"
if mask & HW_SLOWDOWN:
return "ambiguous" # see the nvml.h note: may also fire on a PState change
if mask & OPERATOR:
return "operator"
if mask & BENIGN:
return "benign"
return "clear"
def alert(samples, sustained=3):
"""Page only on a fault, or on HwSlowdown that persists across `sustained` samples."""
cls = [classify(m) for m in samples]
if "fault" in cls:
return True
run = 0
for c in cls:
run = run + 1 if c == "ambiguous" else 0
if run >= sustained:
return True
return False
# --- Happy path: a healthy busy GPU reports nothing ------------------------
assert decode(NONE) == ([], 0)
assert classify(NONE) == "clear"
print(f"mask 0x{NONE:03x}: {classify(NONE)}")
# --- Adversarial 1: an IDLE GPU is "throttled". Any nonzero-means-bad rule pages on it.
names, _ = decode(GPU_IDLE)
assert names == ["GpuIdle"]
assert GPU_IDLE != 0 # a naive `if mask: alert()` fires here
assert classify(GPU_IDLE) == "benign"
assert alert([GPU_IDLE] * 100) is False # never page, however long it persists
print(f"mask 0x{GPU_IDLE:03x}: {classify(GPU_IDLE)} ({names[0]}), a nonzero mask that must not page")
# --- Adversarial 2: real thermal protection, both bits set -----------------
mask = HW_SLOWDOWN | HW_THERMAL_SLOWDOWN
assert mask == 0x48
names, unknown = decode(mask)
assert names == ["HwSlowdown", "HwThermalSlowdown"] and unknown == 0
assert classify(mask) == "fault"
assert alert([mask]) is True # one sample is enough: hardware protection fired
print(f"mask 0x{mask:03x}: {classify(mask)} ({', '.join(names)}), pages on the first sample")
# --- Adversarial 3: HwSlowdown ALONE is ambiguous by NVIDIA's own note -----
# nvml.h: HwSlowdown "May be also reported during PState or clock change".
assert classify(HW_SLOWDOWN) == "ambiguous"
assert alert([HW_SLOWDOWN, NONE, HW_SLOWDOWN]) is False # a blip during a clock change
assert alert([HW_SLOWDOWN] * 3) is True # sustained: real derating
print(f"mask 0x{HW_SLOWDOWN:03x}: {classify(HW_SLOWDOWN)}; blip -> no page, 3 in a row -> page")
# --- Adversarial 4: a future NVML bit must not be silently dropped ---------
# nvml.h on nvmlClocksEventReasonAll: "New reasons might be added to this list in the future".
future = 0x0000000000000200
names, unknown = decode(SW_POWER_CAP | future)
assert names == ["SwPowerCap"] # the known bit still decodes
assert unknown == future # but the unknown one is surfaced, not lost
assert classify(SW_POWER_CAP | future) == "unknown"
print(f"mask 0x{SW_POWER_CAP | future:03x}: known={names}, unknown bits=0x{unknown:03x} (surfaced, not dropped)")
# --- Adversarial 5: the rename is PARTIAL, and value-identical -------------
# nvml.h declares 6 reasons (plus the None sentinel) as nvmlClocksEventReason*, each with
# the old nvmlClocksThrottleReason* name kept as a deprecated alias of the SAME value, and
# 3 hardware reasons never renamed at all. Decoding by value is safe; a blanket sed of the
# identifier ClocksThrottleReason -> ClocksEventReason does not compile.
EVENT_SPELLING = {"GpuIdle": 0x1, "ApplicationsClocksSetting": 0x2, "SwPowerCap": 0x4,
"SyncBoost": 0x10, "SwThermalSlowdown": 0x20,
"DisplayClockSetting": 0x100, "None": 0x0}
THROTTLE_ONLY = {"HwSlowdown": 0x8, "HwThermalSlowdown": 0x40, "HwPowerBrakeSlowdown": 0x80}
# The deprecated ThrottleReason alias of each renamed reason is #defined to the SAME
# value as its EventReason primary, so a mask built from either spelling decodes alike.
DEPRECATED_ALIAS = {"GpuIdle": 0x1, "ApplicationsClocksSetting": 0x2, "SwPowerCap": 0x4,
"SyncBoost": 0x10, "SwThermalSlowdown": 0x20,
"DisplayClockSetting": 0x100, "None": 0x0} # nvmlClocksThrottleReason*
assert DEPRECATED_ALIAS == EVENT_SPELLING # same names -> same bit values
for reason, value in EVENT_SPELLING.items():
assert decode(value) == decode(DEPRECATED_ALIAS[reason]) # alias decodes identically
assert set(EVENT_SPELLING) & set(THROTTLE_ONLY) == set() # no reason carries both spellings
assert all(bit & ALL for bit in THROTTLE_ONLY.values()) # yet all 3 HW bits are in ALL
assert not any(name.startswith("Hw") for name in EVENT_SPELLING) # no EventReasonHw* exists
mixed = EVENT_SPELLING["SwPowerCap"] | THROTTLE_ONLY["HwSlowdown"]
assert decode(mixed)[0] == ["SwPowerCap", "HwSlowdown"] # both spellings in one mask
assert sum(EVENT_SPELLING.values()) | sum(THROTTLE_ONLY.values()) == ALL
print(f"{len(EVENT_SPELLING) - 1} reasons renamed to ClocksEventReason*, "
f"{len(THROTTLE_ONLY)} kept ClocksThrottleReason*; all 9 bits still in ALL=0x{ALL:03x}")
# --- Adversarial 6: a scrape window, decoded in bulk -----------------------
window = np.array([NONE, NONE, SW_POWER_CAP, SW_POWER_CAP | HW_SLOWDOWN,
HW_SLOWDOWN | HW_THERMAL_SLOWDOWN, NONE], dtype=np.uint64)
kinds = [classify(int(m)) for m in window]
assert kinds == ["clear", "clear", "derate", "derate", "fault", "clear"]
assert alert(window.tolist()) is True
busy = np.count_nonzero(window & np.uint64(~ALL & 0xFFFFFFFFFFFFFFFF))
assert busy == 0 # no unknown bits anywhere in the window
print(f"6-sample window -> {kinds}, page={alert(window.tolist())}")
Executed output:
mask 0x000: clear
mask 0x001: benign (GpuIdle), a nonzero mask that must not page
mask 0x048: fault (HwSlowdown, HwThermalSlowdown), pages on the first sample
mask 0x008: ambiguous; blip -> no page, 3 in a row -> page
mask 0x204: known=['SwPowerCap'], unknown bits=0x200 (surfaced, not dropped)
6 reasons renamed to ClocksEventReason*, 3 kept ClocksThrottleReason*; all 9 bits still in ALL=0x1ff
6-sample window -> ['clear', 'clear', 'derate', 'derate', 'fault', 'clear'], page=True
The unknown path is the part most decoders skip. The header says of the All mask: "New reasons might be added to this list in the future". A decoder that iterates its known bits and drops the rest will keep reporting "clear" on a GPU that a newer driver is telling it something new about. Mask with ~ALL and surface the remainder.
How to maintain it: versions, deprecations, and the ABI¶
NVML versions individual functions and structs rather than the library. The header sets NVML_API_VERSION 13 and, by default, maps unsuffixed names onto the newest implementation:
#ifndef NVML_NO_UNVERSIONED_FUNC_DEFS
#define nvmlInit nvmlInit_v2
#define nvmlDeviceGetPciInfo nvmlDeviceGetPciInfo_v3
#define nvmlDeviceGetComputeRunningProcesses nvmlDeviceGetComputeRunningProcesses_v3
#define nvmlEventSetWait nvmlEventSetWait_v2
// ... 16 more, elided
#endif
Define NVML_NO_UNVERSIONED_FUNC_DEFS and the unsuffixed aliases go away, so you must name a version explicitly (nvmlInit_v2). The header states no rationale for the guard, and note it does not by itself pin you to an older surface; it forces you to say which one you mean.3 The change log describes the mechanism: "Functions that changed API and/or size of structs have appended versioning suffix (e.g., nvmlDeviceGetPciInfo_v2). Appropriate C defines have been added that map old function names to the newer version of the function."4
Be careful about how strong a guarantee you infer from that. NVIDIA's own Go bindings assert it ("Since the NVML API is guaranteed to be backwards compatible, we should strive to keep this always up to date with the latest"),7 but the NVML documentation itself never states such a guarantee (it describes the mechanism, not a promise), and its Known Issues page documents a real break: field values numbered 251 to 273 "have changed between 13.0 and 13.0U1/v580TRD2."6 The versioned function and struct surface is stable. Numeric field IDs are not a contract in the same way.
A batch of functions carry DEPRECATED(13.0) in the current header, which on Linux expands to a compiler warning:3
| Deprecated in 13.0 | Replacement |
|---|---|
nvmlDeviceGetCurrentClocksThrottleReasons |
nvmlDeviceGetCurrentClocksEventReasons |
nvmlDeviceGetSupportedClocksThrottleReasons |
nvmlDeviceGetSupportedClocksEventReasons |
nvmlDeviceGetTemperature |
nvmlDeviceGetTemperatureV |
nvmlDeviceGetPowerState |
nvmlDeviceGetPerformanceState ("This function exposes an incorrect generalization") |
nvmlDeviceGetHandleBySerial |
nvmlDeviceGetHandleByUUID |
nvmlDeviceGetApplicationsClock |
None: "Applications clocks are deprecated and will be removed in CUDA 14.0" |
nvmlDeviceGetTemperature is the one that will surprise an exporter maintainer: the function every GPU exporter has called since 2012 is now deprecated in favour of nvmlDeviceGetTemperatureV.
Driver restart recovery¶
A long-running exporter outlives individual driver events: a GPU falling off the bus, an admin-triggered driver reload, or (documented, in nvmlDeviceGetMigMode's own comment) a MIG mode change, which "may require device unbind or reset."3 What must not happen is an exporter that keeps its old handles and its process alive, silently serving the last values it read before the event, because nothing in a plain polling loop forces it to notice.
The documented failure signal is NVML_ERROR_GPU_IS_LOST. nvml.h defines it as firing "if the target GPU has fallen off the bus or is otherwise inaccessible," and it is not confined to query calls: nvmlDeviceRegisterEvents itself lists NVML_ERROR_GPU_IS_LOST among its documented return codes, so the event-registration step of the watch loop, not only the data-read step, can be where a driver-reload-in-progress first surfaces.3 A handle obtained before the loss is not automatically revalidated; nothing in the documented API silently swaps in a fresh one. The exporter's job is to treat this exception as a signal to discard every handle in its device table and re-run nvml_connect()/enumerate_devices() from scratch, which is what run_forever's except pynvml.NVMLError_GpuIsLost branch above does for the scrape path. The event-registration surface needs a second path to the same teardown: it fires inside the background xid_watcher_loop thread, not on run_forever's own call stack, so a bare except in that thread cannot reach the main loop by propagating (a background thread's uncaught exception only kills that thread; nothing carries it to the caller). xid_watcher_loop instead catches it locally and sets a shared gpu_lost_event, which run_forever checks once per iteration alongside its own except branch, so a watcher-detected loss drives the identical disconnect-and-reconnect sequence as a scrape-detected one. An earlier revision of this page only wired the scrape-path branch: an adversarial rerun that killed the watcher's nvmlDeviceRegisterEvents call observed one connect, no disconnect, scrapes continuing regardless, and a watcher thread dead with no signal set anywhere run_forever could see.
nvmlInit_v2 is reference-counted, not idempotent-on-failure. Its own doc comment: "A reference count of the number of initializations is maintained. Shutdown only occurs when the reference count reaches zero."3 That refcounting exists for cooperating callers in one process; it says nothing about what happens to already-issued device handles when the underlying driver goes away and comes back. Calling nvmlShutdown() before every reconnect (as nvml_disconnect() does) is what actually forces the next nvmlInit() to re-establish state, rather than relying on a stale reference count to paper over a driver-side reset.
A newer, more direct signal exists for the unbind/bind transition itself, but this exporter does not use it. Driver 575 and later ship a system-level event API, nvmlSystemEventSetCreate / nvmlSystemRegisterEvents / nvmlSystemEventSetWait, added specifically for this: two bitmask constants, nvmlSystemEventTypeGpuDriverUnbind and nvmlSystemEventTypeGpuDriverBind, are delivered through it, and NVIDIA's change log records the three functions as new additions.[^nvmlsysevent] This is the API that would let an exporter observe "the driver just unbound" and "the driver just rebound" as distinct, first-class events instead of inferring the same fact from a query failing. It requires a driver new enough to expose it and a real bind/unbind transition to fire, neither of which this sandbox has, so it is documented here and not exercised; nvml_exporter.py instead reacts to NVML_ERROR_GPU_IS_LOST on its existing per-device calls, which is the mechanism available on any driver version the rest of this page targets.
What was and was not executed for this section. test_run_forever_wires_http_server_and_xid_watchers_and_recovers (above) is a real, executed test: it monkeypatches scrape_once to raise the real pynvml.NVMLError_GpuIsLost exception once, and asserts that run_forever calls nvml_disconnect exactly once, re-enters nvml_connect/enumerate_devices before resuming, restarts the Xid watchers, and started the real HTTP server exactly once, which is this page's own reconnect and wiring logic running for real against a real exception class, for a loss raised on the scrape path. test_run_forever_recovers_from_watcher_detected_gpu_is_lost is the equivalent check for the other path: a real background xid_watcher_loop thread, not a faked one, hits a real GPU_IS_LOST from nvmlDeviceRegisterEvents, and the test asserts run_forever still reconnects exactly once and keeps scraping, which an adversarial rerun of the page's prior revision failed (one connect, no disconnect). test_sigterm_triggers_clean_shutdown_of_run_forever is the equivalent check for the exit path: a real SIGTERM delivered while the loop is mid-wait must reach the same finally teardown. What was not executed, and cannot be in this sandbox, is a genuine driver unbind/reload on a live GPU: that requires physical hardware and root access to reload the kernel module, which this environment does not have. Treat the recovery logic as validated and the recovery scenario as documented from NVIDIA's own header and change log, not reproduced end to end.
How to run it in production¶
Persistence mode changes what your ECC counters mean. This is not a footnote, it is the difference between a usable and a useless signal. From the header, above the ECC counter enum: "Volatile counts are reset each time the driver loads. On Windows this is once per boot. On Linux this can be more frequent. On Linux the driver unloads when no active clients exist. If persistence mode is enabled or there is always a driver client active (e.g. X11), then Linux also sees per-boot behavior. If not, volatile counts are reset each time a compute app is run."3 On a node without persistence mode, the volatile ECC counter resets between jobs, so an exporter reading it is reporting "errors since the last job started", which is not what any dashboard label claims. Aggregate counts "persist across reboots (i.e. for the lifetime of the device)".3
Alert on the right row-remap field. nvmlDeviceGetRemappedRows returns isPending and failureOccurred. The header is explicit that pending is benign: "A pending remapping won't affect future work on the GPU since error-containment and dynamic page blacklisting will take care of that." failureOccurred "will be set if a row remapping ever failed in the past".3 That is the RMA-grade signal; isPending merely means a reset is needed. (The header states that a remapping failed, and stops there. The usual reading, that the device has run out of spare rows, is an interpretation this page is making, not a sentence NVML prints.) Also note it is unavailable under MIG with active instances.
Check the per-field return code, not just the call. nvmlDeviceGetFieldValues returns NVML_SUCCESS "if any values in values were populated", and each nvmlFieldValue_t carries its own nvmlReturn which "must be checked before looking at value, as value is undefined if nvmlReturn != NVML_SUCCESS".3 An exporter that checks only the top-level status will happily publish uninitialized memory as a metric for every field the device does not support.
The Xid event loop's normal path is a timeout. nvmlEventSetWait returns NVML_ERROR_TIMEOUT when no event arrived, and the docs warn it "in certain conditions can return before specified timeout passes (e.g. when interrupt arrives)".3 Treat timeout as the steady state, not an error. On Linux "every Xid error event would return the associated event data"; on Windows the API coalesces to the last-seen Xid.3
A process manager's stop request is SIGTERM, and an exporter that does not catch it looks the same as one that crashed. run_forever's shutdown_event, wired to SIGTERM/SIGINT by install_signal_handlers above, is what turns a systemctl stop or a Kubernetes pod termination into the same clean-exit path as the reconnect loop's own teardown: the HTTP server's .shutdown() and the Xid watcher threads' stop_event.set() both run before the process exits. Without it, SIGTERM's default action kills the process immediately, mid-scrape or mid-request, which is indistinguishable on a dashboard from an actual crash.
Containers: the NVML failure you will actually hit. The documented symptom is Failed to initialize NVML: Unknown Error, and the cause is not NVML. The NVIDIA Container Toolkit's runtime hook "makes modifications, including setting up cgroup access, to the container without the low-level runtime being aware of these changes", so a later container update (classically a systemctl daemon-reload) strips the GPU device access from a running container and NVML can no longer initialize.10 The documented fixes are to set Docker's cgroup driver to cgroupfs ("exec-opts": ["native.cgroupdriver=cgroupfs"]), to pass the device nodes explicitly, or to use CDI, where "the required device nodes are included in the modifications made to the container config".10 See container toolkit.
Failure modes¶
| Symptom | Cause | Fix |
|---|---|---|
| Dashboards say "GPUs are 95% utilized" but throughput is poor | utilization.gpu is percent of time a kernel was resident, not capacity used3 |
Measure SM/tensor activity via DCGM profiling metrics; treat NVML utilization as a liveness signal only |
| GPU utilization disappeared after enabling MIG | "On MIG-enabled GPUs, querying device utilization rates is not currently supported."3 | Use per-instance DCGM metrics; do not expect NVML utilization under MIG |
| A utilization spike at driver load with no job running | ECC memory scrubbing during driver init3 | Enable persistence mode; ignore the first samples after driver load |
| Volatile ECC counters keep resetting to zero | Without persistence mode, the driver unloads when the last client exits and volatile counts reset per compute app3 | Enable persistence mode; alert on aggregate counters |
| Pager fires on an idle GPU | GpuIdle is a nonzero clock-event reason (executed model above) |
Classify the mask by severity; never treat nonzero as fault |
| Pager fires whenever clocks change | Bare HwSlowdown "May be also reported during PState or clock change"3 |
Require the specific thermal/power-brake bits, or a sustained run of HwSlowdown |
| The exporter publishes plausible garbage for unsupported fields | Only the top-level status was checked; per-field nvmlReturn was not3 |
Check value.nvmlReturn per field before reading value.value |
| Code stops compiling after a driver update | nvmlDeviceGetCurrentClocksThrottleReasons, nvmlDeviceGetTemperature and others are DEPRECATED(13.0)3 |
Move to the *EventReasons and *TemperatureV entry points; values are unchanged |
A blanket rename of ClocksThrottleReason broke the build |
The three hardware reasons were never renamed3 | Keep nvmlClocksThrottleReasonHwSlowdown, ...HwThermalSlowdown, ...HwPowerBrakeSlowdown |
Failed to initialize NVML: Unknown Error in a running container |
Container update stripped cgroup device access set up by the runtime hook10 | native.cgroupdriver=cgroupfs, explicit --device, or CDI injection |
import pynvml gives an API that does not match the docs |
A stale pin of the pynvml package at <12, which shipped its own divergent module8 |
Depend on nvidia-ml-py; from pynvml 12.0.0 the package is only a redirector onto it and emits a FutureWarning |
References¶
- NVIDIA, NVML API Reference Guide (vR610, last updated 2026-05-26): https://docs.nvidia.com/deploy/nvml-api/
- NVIDIA, NVIDIA Management Library (NVML) product page: https://developer.nvidia.com/nvidia-management-library-nvml
- NVIDIA, NVML change log: https://docs.nvidia.com/deploy/nvml-api/change-log.html
- NVIDIA, NVML known issues: https://docs.nvidia.com/deploy/nvml-api/known-issues.html
- NVIDIA,
nvml.has vendored inNVIDIA/go-nvml(NVML_API_VERSION 13; go-nvml states it "is a direct copy ofnvml.hfrom the NVIDIA driver"): https://github.com/NVIDIA/go-nvml/blob/main/gen/nvml/nvml.h - NVIDIA, go-nvml: https://github.com/NVIDIA/go-nvml
- NVIDIA, nvidia-ml-py (the official Python binding): https://pypi.org/project/nvidia-ml-py/
- pynvml on PyPI (deprecated; not the NVML team's binding): https://pypi.org/project/pynvml/
- NVIDIA, DCGM user guide, Getting Started (DCGM sits on top of the driver, NVML and the CUDA Toolkit): https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html
- NVIDIA, Container Toolkit troubleshooting (
Failed to initialize NVML: Unknown Error): https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/troubleshooting.html
Related: nvidia-smi Reference · GPU Diagnostics and Validation · CUPTI: the profiling interface · Observability and Monitoring · Persistence Mode · ECC Support · Reliability, RAS and Failure Modes · Container Toolkit · GPU Software Stack
-
NVIDIA, NVML API Reference (vR610): "The NVIDIA Management Library (NVML) is a C-based programmatic interface for monitoring and managing various states within NVIDIA GPUs." "...intended to be a platform for building 3rd party applications, and is also the underlying library for the NVIDIA-supported nvidia-smi tool." "NVML is thread-safe so it is safe to make simultaneous NVML calls from multiple threads." https://docs.nvidia.com/deploy/nvml-api/nvml-api-reference.html · The product page scopes NVML to "Data Center GPUs" while the API Reference says "NVIDIA GPUs"; the API Reference is the accurate one, since NVML loads on consumer parts with many calls returning
NVML_ERROR_NOT_SUPPORTED. ↩↩ -
NVIDIA, NVML product page: modifiable state is "ECC mode, ECC reset, Compute mode, Persistence mode"; "The NVIDIA Management Library can be downloaded as part of the NVIDIA GPU Driver for Linux and Windows." https://developer.nvidia.com/nvidia-management-library-nvml ↩
-
nvml.h,NVML_API_VERSION 13, read directly from the copy NVIDIA vendors into go-nvml (whose README states the file "is a direct copy ofnvml.hfrom the NVIDIA driver"). Verified in this file: thenvmlUtilization_tfield comments and the "Each sample period may be between 1 second and 1/6 second" note; thenvmlDeviceGetUtilizationRatesMIG and ECC-scrubbing notes; the clock-event reason bit values, withnvmlClocksThrottleReasonHwSlowdown=0x8,nvmlClocksThrottleReasonHwThermalSlowdown=0x40andnvmlClocksThrottleReasonHwPowerBrakeSlowdown=0x80retained under theThrottleReasonspelling while the other six reasons plusNone/Allare primarilynvmlClocksEventReason*with deprecated same-value aliases; theHwSlowdowncause list including "May be also reported during PState or clock change"; "New reasons might be added to this list in the future";nvmlEventData_t.eventDatacarrying the Xid (0 for other events, 999 for unknown);nvmlEventSetWaittimeout and Linux/Windows Xid delivery semantics; thenvmlEccCounterType_tvolatile-vs-aggregate comment;nvmlDeviceGetRemappedRowsisPending/failureOccurredsemantics;nvmlDeviceGetFieldValuesbatching and the per-fieldnvmlReturnrequirement;nvmlDeviceGetSamplesdriver buffering; theNVML_NO_UNVERSIONED_FUNC_DEFSguard and the_vNmapping block; and theDEPRECATED(13.0)markers onnvmlDeviceGetCurrentClocksThrottleReasons,nvmlDeviceGetSupportedClocksThrottleReasons,nvmlDeviceGetTemperature(replaced bynvmlDeviceGetTemperatureV),nvmlDeviceGetPowerState,nvmlDeviceGetHandleBySerialandnvmlDeviceGetApplicationsClock. https://github.com/NVIDIA/go-nvml/blob/main/gen/nvml/nvml.h ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩ -
NVIDIA, NVML change log: "Functions that changed API and/or size of structs have appended versioning suffix (e.g., nvmlDeviceGetPciInfo_v2). Appropriate C defines have been added that map old function names to the newer version of the function." The change log also records, without the page pinning a driver branch to it: "Introduced ClockEventReasons and related APIs which should be used instead of ClockThrottleReasons. Deprecated ClockThrottleReasons." https://docs.nvidia.com/deploy/nvml-api/change-log.html ↩
-
Verified by diffing the
nvml.hthat NVIDIA vendors into go-nvml across release tags: atv0.12.4-0the header declaresNVML_API_VERSION 12and already definesnvmlClocksEventReasonGpuIdle, whilev0.12.1-0(also API version 12) does not; theDEPRECATED(13.0)markers onnvmlDeviceGetCurrentClocksThrottleReasonsand friends appear only fromv0.13.0-0(NVML_API_VERSION 13). So the macro rename and the function deprecation landed in different releases. https://github.com/NVIDIA/go-nvml/blob/main/gen/nvml/nvml.h ↩ -
NVIDIA, NVML known issues (vR610): "NVML Field Values from #251 - #273 (Power Smoothing, Clock Event Reason, and Sync Power Balancing related field values) have changed between 13.0 and 13.0U1/v580TRD2." https://docs.nvidia.com/deploy/nvml-api/known-issues.html ↩
-
NVIDIA, go-nvml README: "These bindings are not a reimplementation of NVML in Go, but rather a set of wrappers around the C API provided by
libnvidia-ml.so." "A working NVIDIA driver withlibnvidia-ml.sois not required to compile code that imports these bindings. However, you will get a runtime error iflibnvidia-ml.sois not available in your library path at runtime." "Thenvml.hfile is a direct copy ofnvml.hfrom the NVIDIA driver. Since the NVML API is guaranteed to be backwards compatible, we should strive to keep this always up to date with the latest." The loader inpkg/nvml/lib.gosetsdefaultNvmlLibraryName = "libnvidia-ml.so.1". https://github.com/NVIDIA/go-nvml ↩↩↩ -
PyPI,
pynvml: "This project has been deprecated. Thepynvmlmodule is NOT developed or maintained in this project! This project provides unofficial NVML Python utilities (i.e. thepynvml_utilsmodule)." "Thepynvml_utilsmodule depends on the official NVML bindings published by NVIDIA under a differentnvidia-ml-pyproject." NVIDIA's own binding isnvidia-ml-py, published by NVIDIA Corporation, and it is the distribution that ships thepynvml.pymodule. Wheel contents show the split by era:pynvml11.5.3 ships its ownpynvml/package, whilepynvml12.0.0 and later ship onlypynvml_utils/plus a_pynvml_redirectorand declare a dependency onnvidia-ml-py. https://pypi.org/project/pynvml/ · https://pypi.org/project/nvidia-ml-py/ ↩↩↩ -
NVIDIA, DCGM user guide: "The user space shared library,
libdcgm.so.4, is the core component of DCGM. This library implements the major underlying functionality and exposes this as a set of C-based APIs. It sits on top of the NVIDIA driver, NVML, and the CUDA Toolkit." NVIDIA publishes no explicit "use DCGM instead of NVML when..." rule; the layering is the only documented relationship. https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/getting-started.html ↩ -
NVIDIA, Container Toolkit troubleshooting: the
Failed to initialize NVML: Unknown Errorsymptom, its cause (the runtime hook "makes modifications, including setting up cgroup access, to the container without the low-level runtime being aware of these changes", so a container update such as asystemctl daemon-reloadremoves GPU access), and the three documented mitigations (native.cgroupdriver=cgroupfs, explicit device nodes, or CDI, where "the required device nodes are included in the modifications made to the container config"). https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/troubleshooting.html ↩↩↩↩