Markdown

The GPU compatibility profile: ownership, qualification, and multi-branch fleets¶

Scope: treating the GPU software stack as one versioned, owned, qualified artefact rather than a set of independently upgradable parts. Covers what belongs in a compatibility profile, who is accountable for it versus who executes changes, the qualification pipeline a profile must pass before production, and how to run two NVIDIA driver branches in one Kubernetes cluster without a node ending up claimed by both. The version facts themselves live on driver versions and branches and CUDA toolkit and runtime; the mechanical upgrade procedure is the rolling driver and CUDA upgrade runbook; fleet-wide pinning and drift are image and config management.

Manifests are reference templates on real APIs, pinned to the versions named in each block and not applied to a cluster during authoring. The Python block was executed; its output is pasted verbatim.

What it is¶

A compatibility profile is a named, versioned tuple describing everything between the silicon and the model, published as an artefact that workloads request by name:

Layer	Examples
Hardware	GPU model and VBIOS, BMC and system firmware, NVSwitch firmware
Fabric	NIC firmware, RDMA stack, switch software
Host	OS release, kernel, node image digest
GPU kernel space	NVIDIA driver branch and exact version, kernel module flavour
Container plumbing	NVIDIA Container Toolkit, device plugin, GPU Operator version
GPU user space	CUDA runtime and toolkit, cuDNN, cuBLAS, NCCL
Framework	PyTorch or JAX build, and what it was compiled against
Serving	engine version, quantisation kernels, model artefact format

The point is not the list. The point is that the tuple has a version number, an owner, a qualification record, and an expiry, so that "is this combination supported here" is a lookup rather than an argument.

Why use it¶

Because every layer has a different team, a different release cadence and a different idea of what "compatible" means, and each change can be individually correct while the composition is broken. The application team picks a newer CUDA runtime. The infrastructure team rolls a driver for a CVE. The hardware team applies a NIC firmware bulletin. Each is defensible. Nobody owns the product of the three.

The compatibility rules themselves are permissive enough to lull you. NVIDIA documents that "backwards compatibility ensures that a newer NVIDIA driver can be used with an older CUDA Toolkit",¹ and that under minor version compatibility, applications built within a CUDA major release "can run, with limited feature-set, on systems having at least the minimum required driver version".² Both are true and both have edges: minor version compatibility carries the requirement of "No PTX (requires SASS)",¹ which means a workload that JITs from PTX falls outside the guarantee that the version numbers suggested it had. Documented compatibility is a starting condition, not a support statement.

Owning the profile is also what makes rollback meaningful. You cannot revert to a known-good state that was never written down.

When to use it (and when not)¶

Formalise a profile when any of these is true:

More than one team can change something in the stack.
The fleet has more than one GPU generation, so one driver branch cannot be optimal for all of it (driver support by tier).
Workloads run as containers against a host driver, so the compatibility surface is a matrix rather than a single install.
Model or engine upgrades ship faster than node images, which is normal.

Do not build the apparatus when the fleet is a handful of identical nodes with one team and one workload. A pinned baseline file and a Makefile are the right size (image and config management). The cost of the profile is real: every additional profile is a combination somebody must keep qualifying.

Architecture¶

flowchart TB
  subgraph OWN["Accountable: GPU platform team"]
    P["Profile vN<br/>firmware + kernel + driver + toolkit + NCCL"]
  end
  subgraph EXEC["Executing teams"]
    HW["Datacentre: BMC, VBIOS, NIC firmware"]
    INF["Infrastructure: node image, kernel"]
    ML["Serving and research: container, engine, model"]
  end
  HW --> P
  INF --> P
  ML -->|"requests a profile, does not define one"| GATE
  P --> GATE["Qualification pipeline<br/>boot, DCGM, NCCL, RDMA, engine, soak"]
  GATE -->|"pass"| CAN["Canary node group"]
  GATE -->|"fail"| BLOCK["Profile not published"]
  CAN --> POOL["Labelled node pools<br/>one active driver stack per node"]
  POOL --> ADM["Admission: workload profile must match node profile"]
  ADM -->|"no qualified profile free"| PEND["Stay pending"]

How to use it: name the owner, then the artefact¶

Several teams execute changes. One team is accountable for the composition. A workable split:

Layer	Executes	Accountable
BMC, VBIOS, system firmware	datacentre or hardware	GPU platform
NIC firmware, switch software, RDMA stack	network	GPU platform
OS, kernel, node image	infrastructure platform	GPU platform
NVIDIA driver, GPU Operator, device plugin, container toolkit	GPU platform	GPU platform
CUDA runtime, cuDNN, NCCL inside the image	serving and research	GPU platform (profile owner)
Engine flags, parallelism, quantisation	model serving	model serving
Model quality and evaluation	research	research
Production release approval	joint change process	service owner

Two rules make the table load-bearing rather than decorative. First, a workload team demonstrates that its model runs on an approved profile; it does not get to introduce one. Second, a hardware or network change reaches production only through the platform team's qualification, however routine the vendor bulletin says it is.

Package user space in the image and leave kernel space on the host. The container carries the CUDA runtime, cuDNN and NCCL; the host provides the qualified kernel driver and container runtime. That is what lets two models with different CUDA runtimes coexist on one driver branch without anyone editing LD_LIBRARY_PATH on a node.

How to develop with it: make the profile checkable¶

A profile that lives in a wiki table is a document. A profile that an admission path evaluates is a control. The model below is the second one. It was executed; the output is pasted verbatim.

# compat_profile.py -- validated: the compatibility profile as an admission decision rather
# than a wiki table. Two checks a fleet running more than one driver branch has to pass:
#   (1) a workload lands only on nodes whose QUALIFIED profile satisfies it, and the
#       satisfies-relation distinguishes NVIDIA's three separate paths rather than calling
#       them all "compatible": ordinary BACKWARD compatibility above the minor-version
#       window, MINOR-VERSION compatibility inside it (which excludes PTX JIT), and the
#       FORWARD-compatibility package below the floor on eligible hardware. Floors are full
#       versions, because comparing branch numbers admits 525.1 for a 525.60.13 floor;
#   (2) driver custom resources partition the fleet: two of them must never select the same
#       node, because a node has exactly one loaded kernel driver stack. Selector text being
#       different is not evidence of that; it has to be tested against real node labels.
# Standard library only.

# NVIDIA's current minor-version-compatibility table is stated in BRANCH numbers, and the
# window has an UPPER bound as well as a floor: above it the application runs through
# ordinary backward compatibility instead, which is a different guarantee carrying no PTX
# restriction. Older release notes stated a Linux patch-level floor for some streams
# (CUDA 12.x was documented as >= 525.60.13 up to the 12.9 notes), so a fleet that pins
# patch levels should carry them here; do not invent one, and re-read the table when a new
# major stream lands. `floor` is compared as a tuple so a patch level can be added without
# changing the comparison.
CUDA_RULES = {
    11: {"floor": (450,), "floor_text": "450", "upper": 525},
    12: {"floor": (525,), "floor_text": "525", "upper": 580},
    13: {"floor": (580,), "floor_text": "580", "upper": None},
}
# Forward compatibility is not a boolean. The compat package targets a specific host driver
# branch range on eligible hardware; an arbitrarily old branch is not covered.
FORWARD_COMPAT_MIN_BRANCH = 450


def parse_version(value, what):
    """Fail closed on anything that is not a dotted numeric version. An admission function
    that throws on a malformed profile is as unhelpful as one that admits it."""
    parts = str(value).split(".")
    if not parts or not all(p.isdigit() for p in parts):
        raise ValueError(f"unparseable {what}: {value!r}")
    return tuple(int(p) for p in parts)


def satisfies(node, need):
    """Does this node's qualified profile run this workload? Returns (ok, reason).
    Every field access is guarded: a malformed WORKLOAD profile must be rejected, not raise."""
    try:
        for field, owner in (("gpu_product", node), ("driver", node)):
            if field not in owner:
                return False, f"node profile missing {field}"
        for field in ("gpu_product", "cuda_runtime"):
            if field not in need:
                return False, f"workload profile missing {field}"
        if not node.get("qualified"):
            return False, "node carries no qualified profile"
        if node["gpu_product"] != need["gpu_product"]:
            return False, f"gpu {node['gpu_product']} != required {need['gpu_product']}"
        major = parse_version(need["cuda_runtime"], "cuda_runtime")[0]
        installed = parse_version(node["driver"], "driver version")
    except ValueError as exc:
        return False, str(exc)

    rule = CUDA_RULES.get(major)
    if rule is None:
        return False, f"CUDA {major}.x has no recorded driver rule"

    if rule["upper"] is not None and installed[0] >= rule["upper"]:
        # Newer than the minor-version window: this is ordinary backward compatibility,
        # which carries no PTX restriction. Naming it correctly matters, because the two
        # paths have different requirements.
        return True, (f"ok under backward compatibility (driver {node['driver']} is newer "
                      f"than the CUDA {major}.x minor-version window)")

    if installed >= rule["floor"]:
        floor = rule["floor_text"]
        if need.get("uses_ptx_jit"):
            # NVIDIA states minor-version compatibility requires SASS and excludes PTX JIT.
            return False, (f"driver {node['driver']} is inside the CUDA {major}.x "
                           f"minor-version window (>= {floor}), which excludes PTX JIT")
        return True, f"ok under minor-version compatibility (>= {floor})"

    floor = rule["floor_text"]
    # Below the floor. The forward-compatibility package is the documented route across a
    # major boundary, and it is a property of the hardware, the package and the host branch.
    if not need.get("forward_compat"):
        return False, (f"driver {node['driver']} below the {floor} floor for CUDA "
                       f"{major}.x, and the workload does not ship forward compatibility")
    if not node.get("forward_compat_eligible"):
        return False, (f"driver {node['driver']} below the {floor} floor and this node is "
                       "not marked eligible for the forward-compatibility package")
    if installed[0] < FORWARD_COMPAT_MIN_BRANCH:
        return False, (f"driver {node['driver']} is below the oldest host branch the "
                       f"compat package supports ({FORWARD_COMPAT_MIN_BRANCH})")
    return True, "ok via the forward-compatibility package"


NODES = {
    "gpu-a": {"gpu_product": "H100", "driver": "580.65.06", "qualified": True,
              "forward_compat_eligible": True, "labels": {"pool": "serve", "branch": "580"}},
    "gpu-b": {"gpu_product": "H100", "driver": "535.183.06", "qualified": True,
              "forward_compat_eligible": True, "labels": {"pool": "train", "branch": "535"}},
    "gpu-c": {"gpu_product": "RTX-6000-Ada", "driver": "535.183.06", "qualified": True,
              "forward_compat_eligible": False, "labels": {"pool": "dev", "branch": "535"}},
    "gpu-d": {"gpu_product": "H100", "driver": "580.65.06", "qualified": False,
              "forward_compat_eligible": True, "labels": {"pool": "serve", "branch": "580"}},
}

# --- (1) Happy path: a CUDA 12.4 H100 workload runs on both H100 nodes, by two DIFFERENT
# routes. 535 sits inside the CUDA 12.x minor-version window; 580 is above its upper bound
# and therefore runs through ordinary backward compatibility, which is a different guarantee.
need_12 = {"gpu_product": "H100", "cuda_runtime": "12.4"}
ok = [n for n, node in NODES.items() if satisfies(node, need_12)[0]]
assert ok == ["gpu-a", "gpu-b"], ok
print(f"1  CUDA 12.4 on H100 -> {ok}")
print(f"1b gpu-b (535): {satisfies(NODES['gpu-b'], need_12)[1]}")
print(f"1c gpu-a (580): {satisfies(NODES['gpu-a'], need_12)[1]}")

# --- (2) Floors are compared as version TUPLES, not as numbers, so a fleet that pins a
# patch level can express it. Swapping in the patch-level floor that NVIDIA documented for
# CUDA 12.x up to the 12.9 release notes changes the verdict for a driver inside the branch.
assert satisfies(dict(NODES["gpu-b"], driver="525.1"), need_12)[0] is True   # branch floor
strict = dict(CUDA_RULES[12], floor=(525, 60, 13), floor_text="525.60.13")
CUDA_RULES[12] = strict
allowed, why = satisfies(dict(NODES["gpu-b"], driver="525.1"), need_12)
assert allowed is False and "525.60.13" in why, why
assert satisfies(dict(NODES["gpu-b"], driver="525.60.13"), need_12)[0] is True
CUDA_RULES[12] = dict(strict, floor=(525,), floor_text="525")
print(f"2  under the branch floor 525, driver 525.1 passes; under a pinned 525.60.13 floor "
      f"it does not: {why}")

# --- (3) Minor-version compatibility excludes PTX JIT. Same node, same runtime, refused the
# moment the workload declares it JITs from PTX, which is the caveat the version numbers hide.
need_ptx = dict(need_12, uses_ptx_jit=True)
allowed, why = satisfies(NODES["gpu-b"], need_ptx)
assert allowed is False and "PTX JIT" in why
# ...but the SAME workload is fine on the newer driver, because that is backward compatibility.
assert satisfies(NODES["gpu-a"], need_ptx)[0] is True
print(f"3  PTX-JIT workload on the 535 node -> rejected: {why}")
print(f"3b the same workload on the 580 node -> {satisfies(NODES['gpu-a'], need_ptx)[1]}")

# --- (4) The rule bites: a CUDA 13.0 build needs >= 580.65.06, so the 535 node drops out.
need_13 = {"gpu_product": "H100", "cuda_runtime": "13.0"}
assert [n for n, nd in NODES.items() if satisfies(nd, need_13)[0]] == ["gpu-a"]
print(f"4  CUDA 13.0 on H100 -> ['gpu-a']; gpu-b: {satisfies(NODES['gpu-b'], need_13)[1]}")

# --- (5) Forward compatibility is the only route across that boundary, and it is reachable
# only below the floor. The same node flips to admissible when the workload ships the package.
need_13_fc = dict(need_13, forward_compat=True)
allowed, why = satisfies(NODES["gpu-b"], need_13_fc)
assert allowed is True and "forward-compatibility" in why
print(f"5  same node, same CUDA 13.0, workload ships the compat package -> {why}")

# --- (6) Adversarial: the package is not a universal solvent. An arbitrarily old host branch
# is outside what it supports, so a boolean eligibility flag is not enough on its own.
ancient = dict(NODES["gpu-b"], driver="418.40.04")
allowed, why = satisfies(ancient, need_13_fc)
assert allowed is False and "oldest host branch" in why, why
print(f"6  same package, host driver 418.40.04 -> rejected: {why}")

# --- (7) Adversarial: eligibility is inventory, not intent. This node is MARKED ineligible,
# and the case proves only that the function respects that marking; deriving eligibility from
# the real supported-SKU list is the part a fleet has to do for itself. NVIDIA scopes the
# package to Data Center GPUs, select NGC Server Ready RTX systems, and Jetson.
need_c = {"gpu_product": "RTX-6000-Ada", "cuda_runtime": "13.0", "forward_compat": True}
allowed, why = satisfies(NODES["gpu-c"], need_c)
assert allowed is False and "not marked eligible" in why
print(f"7  node marked ineligible, package shipped -> rejected: {why}")

# --- (8) Fail closed on an unqualified node. gpu-d is byte-identical to gpu-a except that
# nothing has run the qualification suite on it. It must not be schedulable.
assert satisfies(NODES["gpu-d"], need_12) == (False, "node carries no qualified profile")
print(f"8  gpu-d, identical stack but unqualified -> {satisfies(NODES['gpu-d'], need_12)[1]}")

# --- (9) Adversarial: EVERY malformed input fails closed rather than raising. The first
# version of this model guarded only the node's driver string and threw on a bad workload.
bad_inputs = [
    (NODES["gpu-a"], {"gpu_product": "H100", "cuda_runtime": "R13"}),
    (NODES["gpu-a"], {"gpu_product": "H100", "cuda_runtime": ""}),
    (NODES["gpu-a"], {"cuda_runtime": "12.4"}),
    (dict(NODES["gpu-a"], driver="R580"), need_12),
    ({"qualified": True}, need_12),
    (NODES["gpu-a"], {"gpu_product": "H100", "cuda_runtime": "14.0"}),
]
reasons = []
for node, need in bad_inputs:
    allowed, why = satisfies(node, need)          # must not raise
    assert allowed is False
    reasons.append(why)
print("9  malformed inputs all fail closed, none raise:")
for why in reasons:
    print(f"     {why}")

def selects(selector, labels):
    return all(labels.get(k) == v for k, v in selector.items())


def overlapping(drivers, nodes):
    """Return every (driverA, driverB, node) where two driver resources claim one node."""
    hits = []
    names = sorted(drivers)
    for i, a in enumerate(names):
        for b in names[i + 1:]:
            for node, meta in sorted(nodes.items()):
                if selects(drivers[a], meta["labels"]) and selects(drivers[b], meta["labels"]):
                    hits.append((a, b, node))
    return hits

# --- (10) Two driver branches partitioned by an explicit branch label: no node claimed
# twice, which is the precondition for running both in one cluster at all.
clean = {"d580": {"branch": "580"}, "d535": {"branch": "535"}}
assert overlapping(clean, NODES) == []
print(f"10 selectors keyed on branch -> overlaps {overlapping(clean, NODES)}")

# --- (11) Adversarial: selectors that LOOK disjoint because they use different label keys.
# `pool: serve` and `branch: 535` share no key and no value, so a text diff of the two
# manifests shows nothing in common. On a fleet where a 535 node also serves, both claim
# it. Only evaluating against real node labels finds this.
NODES["gpu-b"]["labels"]["pool"] = "serve"
sneaky = {"d580": {"pool": "serve"}, "d535": {"branch": "535"}}
assert overlapping(sneaky, NODES) == [("d535", "d580", "gpu-b")], overlapping(sneaky, NODES)
print(f"11 disjoint-looking selectors on different keys -> {overlapping(sneaky, NODES)}")

# --- (12) Boundary: a resource with no selector matches every node, so it conflicts with
# any other user-defined resource that selects any node at all.
one_conflict = overlapping({"d580": {}, "d535": {"branch": "535"}}, NODES)
assert [n for _, _, n in one_conflict] == ["gpu-b", "gpu-c"], one_conflict
all_conflict = overlapping({"dall": {}, "dother": {}}, NODES)
assert {n for _, _, n in all_conflict} == set(NODES)
print(f"12 empty selector vs a branch selector conflicts on its {len(one_conflict)} nodes; "
      f"two empty selectors conflict on all {len(all_conflict)}")

print("all assertions passed")

Executed output:

1  CUDA 12.4 on H100 -> ['gpu-a', 'gpu-b']
1b gpu-b (535): ok under minor-version compatibility (>= 525)
1c gpu-a (580): ok under backward compatibility (driver 580.65.06 is newer than the CUDA 12.x minor-version window)
2  under the branch floor 525, driver 525.1 passes; under a pinned 525.60.13 floor it does not: driver 525.1 below the 525.60.13 floor for CUDA 12.x, and the workload does not ship forward compatibility
3  PTX-JIT workload on the 535 node -> rejected: driver 535.183.06 is inside the CUDA 12.x minor-version window (>= 525), which excludes PTX JIT
3b the same workload on the 580 node -> ok under backward compatibility (driver 580.65.06 is newer than the CUDA 12.x minor-version window)
4  CUDA 13.0 on H100 -> ['gpu-a']; gpu-b: driver 535.183.06 below the 580 floor for CUDA 13.x, and the workload does not ship forward compatibility
5  same node, same CUDA 13.0, workload ships the compat package -> ok via the forward-compatibility package
6  same package, host driver 418.40.04 -> rejected: driver 418.40.04 is below the oldest host branch the compat package supports (450)
7  node marked ineligible, package shipped -> rejected: driver 535.183.06 below the 580 floor and this node is not marked eligible for the forward-compatibility package
8  gpu-d, identical stack but unqualified -> node carries no qualified profile
9  malformed inputs all fail closed, none raise:
     unparseable cuda_runtime: 'R13'
     unparseable cuda_runtime: ''
     workload profile missing gpu_product
     unparseable driver version: 'R580'
     node profile missing gpu_product
     CUDA 14.x has no recorded driver rule
10 selectors keyed on branch -> overlaps []
11 disjoint-looking selectors on different keys -> [('d535', 'd580', 'gpu-b')]
12 empty selector vs a branch selector conflicts on its 2 nodes; two empty selectors conflict on all 4
all assertions passed

Five things this pins down.

There are three separate paths, and calling them all "compatible" hides the differences. Case 1 admits the same CUDA 12.4 workload on both H100 nodes by two different routes: driver 535 sits inside the CUDA 12.x minor-version window, while 580 is above its upper bound and therefore runs through ordinary backward compatibility. That distinction is load-bearing, because case 3 shows a PTX-JIT workload refused on the 535 node and admitted on the 580 one. Minor-version compatibility explicitly requires SASS and excludes PTX JIT;¹ backward compatibility carries no such restriction. A profile that records only "CUDA 12.4 is fine here" cannot represent that.

Floors are compared as version tuples, and the documented form has changed. Case 2 is a correction to an earlier version of this page, which asserted patch-level floors of 525.60.13 and 580.65.06. NVIDIA's current table states branch numbers,² and 580.65.06 was never a minor-version minimum at all: it is the driver version packaged with the CUDA 13.0 toolkit, a different table. The patch-level 525.60.13 figure was real, in release notes up to CUDA 12.9, and a fleet that pins patch levels should carry them, which is why the comparison is a tuple. Do not invent one.

Forward compatibility is the only route across a major boundary, and it is not a boolean. Cases 4 and 5 show it reachable only below the floor, which is why the check sits after the floor test. Case 6 is its limit in the other direction: the package targets a supported range of host driver branches, so an arbitrarily old branch is outside it and a single eligibility flag is not enough. Case 7 is the honest framing of hardware eligibility: the node is marked ineligible and the case proves only that the function respects that marking. NVIDIA scopes the package to Data Center GPUs, select NGC Server Ready RTX systems and Jetson boards,³ and deriving eligibility from the real supported-system inventory is work a fleet has to do for itself.

Case 8 is the one people argue about: gpu-d has the same GPU, the same driver and the same everything as gpu-a, and is refused because nothing ran the suite on it. That is the whole discipline in one assertion.

Case 9 covers fail-closed in every direction, which the first version of this model did not: it guarded the node's driver string and threw a KeyError on a malformed workload profile. An unrecorded CUDA major, an unparseable version on either side, and a missing field all now return a rejection with a reason.

Case 11 is the one that bites in practice: two selectors written by two people, sharing no key and no value, that a text review passes and a real fleet fails.

The floors and upper bounds above are NVIDIA's documented minor-version-compatibility ranges: CUDA 11.x needs driver >= 450 and stays inside the window below 525, 12.x needs >= 525 below 580, and 13.x needs >= 580 with no upper bound recorded yet.² Re-read that table when a new major stream lands rather than extrapolating it.

How to run it in production: two branches, one cluster¶

Different driver branches can coexist across node groups in one cluster. They can never coexist on one node: a node has exactly one loaded kernel driver stack.

With the GPU Operator the mechanism is the NVIDIADriver custom resource, one per branch, each pinned to a disjoint node selector. NVIDIA states the constraint directly: "User-defined resources must not select the same node. If they do, the affected resources report notReady with a ConflictingNodeSelector condition, and the Operator retains the existing driver ownership labels until you resolve the conflict."⁴

# nvidiadriver-580.yaml -- reference template, GPU Operator v26.7.0, not applied here.
# Enable with: helm ... --set driver.nvidiaDriverCRD.enabled=true
#              --set driver.nvidiaDriverCRD.deployDefaultCR=false
apiVersion: nvidia.com/v1alpha1
kind: NVIDIADriver
metadata:
  name: serve-580
spec:
  driverType: gpu
  version: "580.65.06"
  nodeSelector:
    nvidia.com/driver-profile: "serve-580"   # disjoint from every other resource
---
apiVersion: nvidia.com/v1alpha1
kind: NVIDIADriver
metadata:
  name: train-550
spec:
  driverType: gpu
  version: "550.90.07"
  nodeSelector:
    nvidia.com/driver-profile: "train-550"

Three edges worth writing on the runbook rather than discovering:

A user-defined NVIDIADriver without a node selector matches every GPU node. It may still coexist with the chart's default resource, which it takes precedence over, but it conflicts with any other user-defined resource selecting any of the same nodes.⁴ Case 9 above is that rule as an assertion.
Only one resource may set spec.default: true, and a default resource cannot carry a node selector, because a fallback that only matches some nodes is not a fallback.⁴
Use one label key that means "which driver profile is this node qualified for", and select on that key alone. Selecting on incidental labels such as a pool name is how case 8 happens.
These resources compose with ClusterPolicy rather than replacing it. NVIDIA's instruction is to set spec.driver.useNvidiaDriverCRD to true on the cluster policy in order to use driver custom resources with it; the Operator does not reconcile driver resources while that field is false.

If you instead keep drivers on the host image, that is a legitimate choice, and the boundary must be explicit: "The GPU Operator only manages the lifecycle of containerized drivers. Drivers which are pre-installed on the host are not managed by the GPU Operator."⁵ Set driver.enabled=false and let the Operator own the device plugin, toolkit integration, feature discovery and DCGM instead.⁶ What must not happen is configuration management, an image pipeline and the Operator all believing they own the same driver.

The qualification pipeline¶

A profile is published only after it passes, in order, a suite whose failures localise:

Node boots on the target image; driver and kernel modules load; nvidia-smi enumerates every expected GPU (driver and module load failure is the failure path).
Fabric Manager and NVSwitch come up where applicable (Fabric Manager).
dcgmi diag at the level appropriate to the change: the quick level as a gate, the long level on a drained node for a driver or firmware change (diagnostics and validation has the level table).
CUDA execution, GPU memory and ECC checks (ECC support).
Intra-node collectives, then inter-node collectives, against a stored baseline for that exact topology, not against a universal bandwidth number (fabric performance regression).
RDMA and GPUDirect validated independently of NCCL, so a network fault is not read as a collective bug.
Container runtime: a GPU container starts, sees its devices, and runs a kernel (container toolkit and CDI).
The real serving or training workload starts and produces representative latency and throughput.
Soak under sustained load, because thermal and memory regressions do not appear in a sixty-second benchmark.
Failure and recovery: drain, reboot, and node loss behave as the runbooks say they do.

Roll the qualified profile through canary node groups, one failure domain at a time, comparing performance and error rate with the incumbent pool. A driver update is a disruptive node operation by construction: the Operator's own upgrade sequence requires you to "disable all clients to the GPU driver", unload the modules, start the updated pod, install and load the new modules, and re-enable clients.⁵ Treat it as maintenance with a rollback path, not as a rolling app deploy.

Keep at least the previous known-good node image and application profile. New and old may coexist in separate node groups for a window; labels and taints are what stop a workload landing on the wrong side of that boundary.

How to maintain it¶

Give every profile an expiry. Without one, the fleet accumulates a branch per model and the qualification matrix becomes the team's whole job.
Cap the number of live profiles and make adding one a decision with a named owner, not a side effect of a deployment.
Re-derive the matrix on every major stream. The minimum-driver table changes when a new CUDA major lands; case 5 above is the fail-closed behaviour that buys you time to update it.
Do not read nvidia-smi as an inventory of installed toolkits. The CUDA version it prints is the maximum the driver supports: the NVIDIA driver "reports a maximum version of CUDA supported and thus is able to run applications built with CUDA Toolkits up to that version."⁷
Record firmware separately. Application rollback is a deployment operation; firmware rollback is a hardware recovery operation, may be unsupported by the vendor, and needs quarantine capacity budgeted before the rollout starts (GSP firmware and driver mismatch).
Re-qualify after any fabric change. NIC and switch firmware sit inside the profile even though they are nobody's idea of GPU software.

Failure modes¶

Two NVIDIADriver resources claiming one node. Selectors written independently; both go notReady with ConflictingNodeSelector and the node keeps its existing driver.
A workload lands on an unqualified node because admission checked a GPU count rather than a profile. It usually works, until the one workload that needs the thing nobody tested.
Version numbers pass, the workload does not. Minor version compatibility excludes PTX JIT; a build that relies on it is outside the guarantee its version numbers implied.
The Operator and the host image both believe they own the driver. The Operator's own behaviour here is deterministic rather than a race: its driver-manager init container detects a pre-installed host driver, relabels the node nvidia.com/gpu.deploy.driver=pre-installed, and the driver pod terminates and is not rescheduled because the DaemonSet's node selector no longer matches. The real hazard is what happens afterwards, when an out-of-band host change alters a driver the Operator has stopped tracking and the fleet drifts into two populations with nothing reconciling them.
A firmware rollout with no rollback plan. Vendor downgrade is unsupported, the cohort is quarantined, and the quarantine is larger than the spare capacity.
The oldest GPU generation pins the whole fleet's branch. Real and unavoidable in a mixed estate; the answer is separate profiles per pool, not one compromise branch (driver support by tier).
nvidia-smi read as proof a toolkit is installed. It reports a driver capability, not an installation.
The qualification suite passes but nothing recorded the baseline. A later regression has nothing to be a regression against.

References¶

NVIDIA CUDA compatibility, overview: https://docs.nvidia.com/deploy/cuda-compatibility/latest/index.html
Why CUDA compatibility (backward, minor version, forward): https://docs.nvidia.com/deploy/cuda-compatibility/latest/why-cuda-compatibility.html
CUDA minor version compatibility and minimum driver table: https://docs.nvidia.com/deploy/cuda-compatibility/latest/minor-version-compatibility.html
CUDA forward compatibility (the cuda-compat package and the eligible hardware list): https://docs.nvidia.com/deploy/cuda-compatibility/latest/forward-compatibility.html
CUDA Toolkit, driver and architecture matrix (what nvidia-smi reports): https://docs.nvidia.com/datacenter/tesla/drivers/cuda-toolkit-driver-and-architecture-matrix.html
NVIDIA GPU Operator overview: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/index.html
GPU Operator driver upgrades (upgrade sequence, upgrade-state labels): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html
GPU Operator driver configuration (NVIDIADriver custom resource): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-configuration.html
GPU Operator getting started (driver.enabled=false for pre-installed drivers): https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html

NVIDIA CUDA compatibility: "Backwards compatibility ensures that a newer NVIDIA driver can be used with an older CUDA Toolkit"; the same page's matrix lists the minor-version path with the requirement "No PTX (requires SASS), NVCC target architecture required." https://docs.nvidia.com/deploy/cuda-compatibility/latest/why-cuda-compatibility.html ↩↩↩
NVIDIA CUDA minor version compatibility: "From CUDA 11 onwards, applications compiled with a CUDA Toolkit release from within a CUDA major release family can run, with limited feature-set, on systems having at least the minimum required driver version as indicated below." The accompanying table gives CUDA 13.x >= 580, 12.x >= 525, 11.x >= 450. https://docs.nvidia.com/deploy/cuda-compatibility/latest/minor-version-compatibility.html ↩↩↩
NVIDIA CUDA forward compatibility: the package is scoped to "NVIDIA Data Center GPUs", "Select NGC Server Ready SKUs of RTX cards" and Jetson boards, and exists "to support applications built on newer CUDA Toolkits to run on systems installed with an older NVIDIA Linux GPU driver from different major release families." https://docs.nvidia.com/deploy/cuda-compatibility/latest/forward-compatibility.html ↩
NVIDIA GPU Operator driver configuration: "User-defined resources must not select the same node. If they do, the affected resources report notReady with a ConflictingNodeSelector condition, and the Operator retains the existing driver ownership labels until you resolve the conflict." The same page states that only one resource may set spec.default: true, that a default resource cannot specify a node selector, and that a user-defined resource without a node selector matches all GPU nodes. https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-configuration.html ↩↩↩
NVIDIA GPU Operator driver upgrades: "The GPU Operator only manages the lifecycle of containerized drivers. Drivers which are pre-installed on the host are not managed by the GPU Operator." The documented upgrade sequence is to disable all clients to the GPU driver, unload the current kernel modules, start the updated driver pod, install and load the updated modules, and re-enable clients. https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html ↩↩
NVIDIA GPU Operator getting started: --set driver.enabled=false "prevents the Operator from installing the GPU driver on any nodes in the cluster", and is the documented setting for systems with pre-installed drivers. https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html ↩
NVIDIA CUDA Toolkit, driver and architecture matrix: "when using tools such as nvidia-smi, the NVIDIA driver reports a maximum version of CUDA supported and thus is able to run applications built with CUDA Toolkits up to that version." https://docs.nvidia.com/datacenter/tesla/drivers/cuda-toolkit-driver-and-architecture-matrix.html ↩