Markdown

AI-assisted performance optimization¶

Scope: using AI to optimize AI systems: LLM/agent-driven kernel generation and autotuning, AI-discovered algorithms (AlphaTensor-style faster GEMM), automated profile-to-fix loops, and AI-assisted cluster operations; where a human engineer is still required; plus how to keep every result verified.

Reference-template note: the flags, APIs, GPUs, and speedup numbers on this page are from the cited book chapter and official NVIDIA/DeepMind/Predibase/Stanford sources; nothing here has been hardware-tested in this knowledge base. The numeric code blocks below are runnable and self-checking (they assert their own results), but they validate the underlying math, not any specific GPU kernel. Validate every generated kernel and every claimed speedup on your own target GPU before relying on it.

What it is¶

A set of techniques where AI systems generate, tune, or operate the low-level code and infrastructure that AI workloads run on. Four distinct layers, ordered from narrowest to broadest:

AI-discovered algorithms. Search over the mathematical structure of an operation, not its code. DeepMind's AlphaTensor reformulated the search for fast matrix-multiplication algorithms as a single-player reinforcement-learning game (AlphaZero-derived), exploring tensor decompositions a human could never enumerate. It rediscovered Strassen's subquadratic 2x2 algorithm and improved on it for larger sizes, and found a decomposition tuned to the NVIDIA V100 that multiplied large matrices 10 to 20% faster than the standard V100-era cuBLAS at the time (DeepMind, Nature 2022).
LLM/agent-driven kernel generation. A reasoning LLM writes a GPU kernel (CUDA or Triton) inside a closed loop with a verifier that checks correctness and measures runtime, feeding feedback back as a refined prompt. NVIDIA ran DeepSeek-R1 on an H100 with a roughly 15-minute inference-time-scaling budget per problem to generate attention kernels, reaching 1.1x to 2.1x over PyTorch FlexAttention and 100% (Level-1) / 96% (Level-2) numerical correctness on Stanford's KernelBench (NVIDIA Technical Blog). The same closed loop extends to compiler autoscheduling: ComPilot (Merouani et al., PACT 2025) has an LLM propose loop transformations (tiling, fusion, interchange) to a compiler that grounds each proposal with a legality check and a runtime measurement, iterating zero-shot to geometric-mean speedups of 2.66x (single run) and 3.54x (best-of-5) on PolyBench, competitive with the Pluto polyhedral optimizer without task-specific fine-tuning. The generator differs (loop schedules rather than kernel source), but the grounded generate-verify-refine loop is identical.
RL-fine-tuned kernel autotuning. Instead of prompting a frozen model, fine-tune one with reinforcement learning so kernel quality is the reward. Predibase fine-tuned Qwen2.5-Coder-32B-Instruct with GRPO (Group Relative Policy Optimization) on H100s, rewarding kernels that compile, produce correct results, and beat a baseline (Predibase RFT). Fregly reports the model reached working Triton kernels roughly 40% of the time after around 5,000 steps (from near-0%), with some kernels up to 3x baseline (Fregly, Ch. 20).
AI-assisted cluster operations. AI moves from writing code to running the cluster: anomaly detection on training/inference telemetry, automated failure analysis from logs and device stats, and RL-based real-time control of knobs (power/frequency, cache eviction, congestion control, memory swap policy) that have too many interacting dimensions for manual tuning (Fregly, Ch. 20). The full incident lifecycle an agent runs (detect, localize, RCA, mitigate, and confirm recovery) is covered in Agentic AIOps and Autonomous Operations.

The common thread across all four layers is the generate -> verify -> refine loop: a generator (a prompted model, an RL policy, or an algorithm search) proposes a candidate, a verifier decides if it is both correct and faster, and the verdict feeds back into the next proposal. The generator is interchangeable; the verifier is what makes the loop trustworthy.

Why use it¶

The verified gains are first-class, not marginal. A 10 to 20% GEMM speedup is "free compute" on every forward and backward pass of every model, recovered through smarter software alone, no new hardware (DeepMind, Nature 2022). These are the same magnitudes of improvement a new GPU generation buys, or weeks of expert CUDA tuning.

The ROI on kernel generation is about engineer time. A CUDA expert may spend hours or days hand-writing and testing a new attention-kernel variant; the AI-in-a-loop produced a comparable kernel in roughly 15 minutes, freeing engineers for the higher-level optimization and edge cases AI handles poorly (Fregly, Ch. 20). At datacentre scale the operational layer matters as much: an AI scheduler that colocates a compute-heavy and a bandwidth-heavy job, or that flags ECC errors on a node before a three-month run crashes, directly raises goodput (useful neural compute per unit time), the metric that actually maps to cost, not raw "GPU 100% busy."

This is also a codesign story: the headline result repeated across the book's case studies is that real LLM performance comes from tightly integrated hardware/software/algorithm optimization, and AI now participates at all three of those levels.

When to use it (and when not)¶

Reach for AI-assisted optimization when:

A hot kernel or operation has a clear, machine-checkable correctness oracle (reference output) and a measurable runtime: attention variants, GEMM, elementwise/fused ops. The verifier is what makes the loop trustworthy.
The search space is too large for human enumeration: numerical-precision choices, launch-parameter autotuning, algorithmic decompositions, or cluster-scheduling policies with many interacting knobs.
You are operating a large fleet where 24/7 telemetry triage (anomaly detection, root-cause from logs) exceeds human attention budgets.

A human engineer is still required when:

There is no cheap, reliable verifier. The loop is only as safe as its oracle. NVIDIA's own guidance is to back the generator with a solid test suite (KernelBench-style) so rare edge cases do not slip through: Level-2 correctness was 96%, not 100% (NVIDIA Technical Blog). The remaining few percent is exactly where humans must review.
Results are unvalidated or unmerged. As of the book's writing, AlphaTensor's matrix-multiply algorithms remained experimental and were not incorporated into mainstream libraries like cuBLAS, pending further validation and generalization (Fregly, Ch. 20). Treat AI-discovered algorithms as candidates, not drop-in replacements.
The situation is novel. The engineer sets objectives, safety and fairness guardrails, and handles cases the AI has not seen; routine load-balancing, failure recovery, and buffer tuning are delegated (Fregly, Ch. 20). The analogy in the book is autopilot: deep oversight required, millisecond control automated.
Forward-looking autonomy. Fully self-improving agents, perpetual continual learning (catastrophic-forgetting risk), and fully automatic pipeline parallelism are described in the book as active research / hypothetical projections, not shipping capability. Current frameworks still require explicit pipeline-parallel implementations (Fregly, Ch. 20). Date-sensitive; do not architect production systems around them.

Architecture¶

Every layer instantiates the same closed loop. A generator proposes a candidate kernel or algorithm; the verifier compiles it, checks it against a trusted reference oracle, and times it against an explicit baseline; only a verdict of correct and fast enough accepts the candidate; otherwise the feedback (compile error, numeric diff, or measured slowdown) is folded back into the next proposal, either as a refined prompt (frozen model) or as a reward gradient (RL fine-tuning).

flowchart LR
  P["Prompt / task spec"] --> G["Generator<br/>frozen LLM, RL policy,<br/>or algorithm search"]
  G --> C["Compile / build<br/>on target GPU"]
  C -->|"build fails"| F["Refine prompt / reward"]
  C -->|"builds"| V["Verifier<br/>correctness vs reference oracle"]
  V -->|"numerically wrong"| F
  V -->|"correct"| T["Timing vs explicit baseline"]
  T -->|"not faster"| F
  T -->|"correct and fast enough"| A["Accept + anchor proof<br/>(reference, hardware, baseline)"]
  F --> G
  A --> H{"Human review gate<br/>on residual failure band"}
  H -->|"pass"| PROD["Promote to production"]
  H -->|"reject"| G

The load-bearing edge is V -> F on "numerically wrong": a silent correctness bug that reaches production is far worse than a slow kernel, so the oracle must be exact, not approximate. The rest of this page shows how to build each box and, critically, how to make the verifier trustworthy.

How to build the loop¶

The non-negotiable component is the verifier, not the generator. Stand up: (a) a reference implementation for correctness, (b) a compile/run harness on the target GPU (results are hardware-specific: AlphaTensor's win was V100-specific), and (c) a timing measurement against an explicit baseline. Only a verdict of "correct and faster" should accept a candidate. Pin the generation budget (NVIDIA used roughly 15 min/problem of inference-time scaling) and an iteration cap.

The loop itself is small. This runnable model uses a real correctness oracle (equivalence to a slow triple-loop reference) plus timing, and proves the property that matters: a subtly wrong kernel (an off-by-one on the contraction dimension) is rejected, and the loop only accepts a correct candidate. Run with the system python3 (numpy 2.x):

import time
import numpy as np


def reference_gemm(a, b):
    """Trusted-but-slow oracle: triple loop, the ground truth for correctness."""
    m, k = a.shape
    _, n = b.shape
    out = np.zeros((m, n), dtype=a.dtype)
    for i in range(m):
        for j in range(n):
            acc = a.dtype.type(0)
            for p in range(k):
                acc += a[i, p] * b[p, j]
            out[i, j] = acc
    return out


def verify(candidate, trials=5, atol=1e-8):
    """Compile+correctness+timing verdict against the reference oracle."""
    best = float("inf")
    rng = np.random.default_rng(0)
    for _ in range(trials):
        a = rng.standard_normal((16, 24))
        b = rng.standard_normal((24, 20))
        ref = a @ b  # fast trusted equivalent of reference_gemm, used as oracle
        try:
            got = candidate(a, b)
        except Exception as exc:                       # a kernel that will not run
            return False, float("inf"), f"raised {type(exc).__name__}"
        if got.shape != ref.shape or not np.allclose(got, ref, atol=atol, rtol=1e-6):
            return False, float("inf"), "incorrect"
        t0 = time.perf_counter()
        candidate(a, b)
        best = min(best, time.perf_counter() - t0)
    return True, best, "correct"


def _broken(a, b):        raise RuntimeError("undefined symbol")
def _subtly_wrong(a, b):  return a[:, :-1] @ b[:-1, :]   # off-by-one on the contraction dim
def _slow_correct(a, b):  return np.array([[float(np.dot(a[i], b[:, j]))
                                            for j in range(b.shape[1])]
                                            for i in range(a.shape[0])])
def _fast_correct(a, b):  return a @ b


CANDIDATES = [_broken, _subtly_wrong, _slow_correct, _fast_correct]


def run_loop(max_iters, target_time):
    """Return (accepted_iteration, accepted). -1 iteration => never accepted."""
    for iteration in range(max_iters):
        candidate = CANDIDATES[min(iteration, len(CANDIDATES) - 1)]
        valid, runtime, _ = verify(candidate)          # compile + correctness + timing
        if valid and runtime <= target_time:
            return iteration, True
        # else: fold feedback into the next prompt/reward (elided) and try again
    return -1, False


# Happy path: with a generous budget the loop accepts the first CORRECT kernel.
it, ok = run_loop(max_iters=8, target_time=1.0)
assert ok, "loop should accept a correct kernel"
assert it == 2, f"first correct kernel is _slow_correct at iter 2, got {it}"

# Adversarial 1: the off-by-one kernel is a SILENT numeric bug and must be rejected.
valid_wrong, _, fb = verify(_subtly_wrong)
assert valid_wrong is False and fb == "incorrect"

# Adversarial 2: a kernel that does not run is rejected, not allowed to crash the loop.
valid_broken, _, _ = verify(_broken)
assert valid_broken is False

# Adversarial 3: an impossible time budget => loop exhausts iters without accepting.
it2, ok2 = run_loop(max_iters=3, target_time=-1.0)
assert ok2 is False and it2 == -1

# Equivalence: the slow reference and the fast path agree (guards the oracle itself).
a = np.random.default_rng(0).standard_normal((7, 5))
b = np.random.default_rng(1).standard_normal((5, 9))
assert np.allclose(reference_gemm(a, b), a @ b, atol=1e-9)

print("loop OK: accept iter", it, "| wrong rejected | broken rejected | no false-accept")

Choose the technique by what you have. Frozen reasoning model plus verifier loop (the NVIDIA/DeepSeek-R1 pattern) when you have a strong model and a good oracle but no training budget. RL fine-tuning (the Predibase/GRPO pattern, see GRPO) when you have many examples and want a specialized, smaller model whose reward is correctness-and-speed. Algorithm search (the AlphaTensor pattern) only for fundamental operations where a mathematical reformulation is plausible. That last one is research-grade effort.

How the algorithm search works (AlphaTensor)¶

AlphaTensor does not search over kernel code; it searches over the mathematical structure of matrix multiplication itself. The n x n matmul is a fixed 3D tensor T, and any way to write it as a sum of R rank-1 terms (a CP decomposition into factors U, V, W) is an algorithm that uses exactly R scalar multiplications. Fewer terms means a faster algorithm. Strassen's classic 2x2-in-7 scheme is the canonical rank-7 solution, strictly better than the trivial rank-8 (eight multiplies). AlphaTensor learned to find such decompositions, and low-rank ones tuned to specific hardware, by self-play.

This numpy-only block reconstructs the matmul tensor from Strassen's factors, runs the resulting bilinear algorithm, and asserts it equals standard matmul on 200 random inputs, that it uses 7 multiplies (not 8), and that corrupting a single factor entry is detected (the search would reject it). Runnable with system python3:

import numpy as np


def matmul_tensor(n):
    """The (n^2, n^2, n^2) tensor of n x n matmul: C[i,j] = sum_l A[i,l] B[l,j]."""
    T = np.zeros((n * n, n * n, n * n), dtype=np.int64)
    for i in range(n):
        for j in range(n):
            for l in range(n):
                T[i * n + l, l * n + j, i * n + j] = 1   # (A index, B index, C index)
    return T


# Strassen's rank-7 factors for 2x2, columns = the 7 scalar products.
# Derived from M1=(a00+a11)(b00+b11), M2=(a10+a11)b00, M3=a00(b01-b11),
# M4=a11(b10-b00), M5=(a00+a01)b11, M6=(a10-a00)(b00+b01), M7=(a01-a11)(b10+b11).
U = np.array([[1, 0, 1, 0, 1, -1, 0],
              [0, 0, 0, 0, 1, 0, 1],
              [0, 1, 0, 0, 0, 1, 0],
              [1, 1, 0, 1, 0, 0, -1]], dtype=np.int64)     # linear combos of A entries
V = np.array([[1, 1, 0, -1, 0, 1, 0],
              [0, 0, 1, 0, 0, 1, 0],
              [0, 0, 0, 1, 0, 0, 1],
              [1, 0, -1, 0, 1, 0, 1]], dtype=np.int64)      # linear combos of B entries
W = np.array([[1, 0, 0, 1, -1, 0, 1],
              [0, 0, 1, 0, 1, 0, 0],
              [0, 1, 0, 1, 0, 0, 0],
              [1, -1, 1, 0, 0, 1, 0]], dtype=np.int64)      # combine products into C


def reconstruct(U, V, W):
    """T_hat[a,b,c] = sum_r U[a,r] V[b,r] W[c,r]  (rank-R CP decomposition)."""
    return np.einsum("ar,br,cr->abc", U, V, W)


def apply_bilinear(U, V, W, A, B):
    """Run the R-multiplication algorithm the factors define."""
    left = U.T @ A.reshape(-1)          # R linear combos of A
    right = V.T @ B.reshape(-1)         # R linear combos of B
    m = left * right                    # exactly R scalar multiplications
    return (W @ m).reshape(A.shape)


T = matmul_tensor(2)
R = U.shape[1]

# 1. The factors are a valid decomposition of the exact matmul tensor.
assert np.array_equal(reconstruct(U, V, W), T)

# 2. Rank / multiplication count: 7 products, strictly fewer than the trivial 2^3 = 8.
assert R == 7 and R < 2 ** 3

# 3. The reconstructed algorithm equals standard matmul (equivalence to slow reference).
rng = np.random.default_rng(7)
for _ in range(200):
    A = rng.integers(-4, 5, (2, 2))
    B = rng.integers(-4, 5, (2, 2))
    assert np.array_equal(apply_bilinear(U, V, W, A, B), A @ B)

# 4. Adversarial corruption: flip ONE factor entry => tensor identity breaks AND
#    the algorithm produces a detectably wrong product on some input.
Ubad = U.copy(); Ubad[0, 0] += 1
assert not np.array_equal(reconstruct(Ubad, V, W), T)
mismatch = False
for _ in range(50):
    A = rng.integers(-4, 5, (2, 2)); B = rng.integers(-4, 5, (2, 2))
    if not np.array_equal(apply_bilinear(Ubad, V, W, A, B), A @ B):
        mismatch = True
        break
assert mismatch, "a corrupted factor must yield a detectably wrong product"

print("AlphaTensor core OK: rank", R, "< 8 | tensor identity holds | 200/200 exact | corruption caught")

The point this proves is exactly why the book keeps AlphaTensor's results out of production libraries: a decomposition either reproduces the matmul tensor exactly or it does not, and the only way to know is to check it, on real inputs, on the target hardware.

How the RL reward works (GRPO autotuning)¶

Predibase's fine-tuning path replaces the frozen model with one trained by GRPO, so the reward is kernel quality. GRPO samples a group of G candidate kernels per problem and scores each one's advantage relative to its group's mean, with no learned critic. The reward is staged: no credit unless the kernel compiles and is correct, then a bonus for beating the baseline. This numpy-only block validates the two pieces of core math (the staged reward and the group-relative advantage), checks the advantage against a slow reference on 500 random groups, and handles the adversarial zero-variance group (every kernel identical) that would divide by zero without the epsilon guard. Runnable with system python3:

import numpy as np


def kernel_reward(compiles, correct, speedup):
    """Staged reward: correctness gate dominates, then reward speed.

    speedup = baseline_time / candidate_time; > 1 means faster than baseline.
    """
    if not compiles:
        return 0.0
    if not correct:
        return 0.1                              # ran but wrong: tiny partial credit
    return 1.0 + max(0.0, speedup - 1.0)        # correct, plus bonus for beating baseline


def group_advantages(rewards, eps=1e-8):
    """GRPO advantage: (r - mean) / (std + eps) within the group (vectorised)."""
    return (rewards - rewards.mean()) / (rewards.std() + eps)


def group_advantages_reference(rewards, eps=1e-8):
    """Slow explicit reference the vectorised version must match exactly."""
    n = len(rewards)
    mean = sum(rewards) / n
    std = (sum((r - mean) ** 2 for r in rewards) / n) ** 0.5
    return [(r - mean) / (std + eps) for r in rewards]


# 1. Reward staging: the correctness gate dominates raw speed.
assert kernel_reward(compiles=False, correct=True, speedup=9.0) == 0.0
assert kernel_reward(compiles=True, correct=False, speedup=9.0) == 0.1
assert kernel_reward(True, True, 3.0) > kernel_reward(True, True, 1.0)

# 2. Vectorised advantage equals the slow reference (equivalence) on 500 random groups.
rng = np.random.default_rng(20)
for _ in range(500):
    g = rng.standard_normal(rng.integers(2, 12)) * rng.uniform(0.1, 5)
    assert np.allclose(group_advantages(g), group_advantages_reference(g.tolist()), atol=1e-9)

# 3. Group-relative property: advantages are mean-centred and the best kernel wins.
group = np.array([kernel_reward(True, True, s) for s in (1.0, 1.5, 4.0, 1.0)])
adv = group_advantages(group)
assert abs(adv.sum()) < 1e-6
assert int(np.argmax(adv)) == int(np.argmax(group)) == 2      # the 4x kernel

# 4. Adversarial edge case: a zero-variance group (all kernels identical) must stay
#    finite and give ~0 signal, not divide by zero.
flat = np.full(6, 1.0)
adv_flat = group_advantages(flat)
assert np.all(np.isfinite(adv_flat)) and np.allclose(adv_flat, 0.0, atol=1e-6)

# 5. Adversarial: an all-failed group (nothing compiles) is also finite.
assert np.all(np.isfinite(group_advantages(np.zeros(5))))

print("GRPO core OK: reward gating | 500/500 advantage==reference | argmax=2 | zero-var group ~0")

The reference template below is the real training call that this reward math plugs into. It needs TRL and torch (not installed here, so it is a reference template, not executed); the numpy block above is the executable validation of its reward core:

# Reference template only: requires `trl`, `torch`, and multiple H100s (not run here).
# The numpy block above validates the reward + advantage math this trainer uses.
from trl import GRPOConfig, GRPOTrainer  # pip install trl


def reward_kernels(completions, **kwargs):
    """One reward per sampled kernel: compile -> correctness vs oracle -> speedup bonus."""
    rewards = []
    for code in completions:
        compiled = try_compile(code)            # your Triton/CUDA build harness
        if not compiled:
            rewards.append(0.0); continue
        correct = matches_reference(compiled)   # exact check vs a trusted oracle
        if not correct:
            rewards.append(0.1); continue
        speedup = baseline_time() / measured_time(compiled)   # timed on the TARGET GPU
        rewards.append(1.0 + max(0.0, speedup - 1.0))
    return rewards


trainer = GRPOTrainer(
    model="Qwen/Qwen2.5-Coder-32B-Instruct",     # Predibase's base model
    reward_funcs=reward_kernels,                  # GRPO groups + normalises internally
    args=GRPOConfig(num_generations=8, max_steps=5000),
    train_dataset=kernel_problems,                # e.g. KernelBench tasks
)
trainer.train()   # Fregly, Ch. 20: ~40% working Triton kernels after ~5,000 steps

How to integrate with the toolchain¶

Integrate with the existing toolchain, do not replace it. AI-generated kernels still flow through the same compilers and runtimes: Triton compiles Python-like kernels to optimized code per GPU architecture, CUTLASS backs hand- and tool-written GEMM, and smart compilers already automate kernel fusion, launch-parameter autotuning, and graph capture (CUDA Graphs via cudaGraphInstantiate() / cudaGraphLaunch() reduce per-launch CPU overhead). The book's framing: the engineer's role shifts toward guiding the tools and quickly verifying they did a good job rather than hand-tuning every knob.

A generated kernel is the sort of artifact the loop above emits. This is what the generator box produces for the Triton path (it needs triton and torch, not installed here, so it is a reference template; the loop and oracle that would accept or reject it are the executed numpy block under "How to build the loop"):

# Reference template only: requires `triton` and `torch` on a CUDA GPU (not run here).
# The generate->verify->refine loop (executed numpy block above) is what would gate this.
import triton
import triton.language as tl


@triton.jit
def add_kernel(x_ptr, y_ptr, out_ptr, n, BLOCK: tl.constexpr):
    pid = tl.program_id(0)
    offs = pid * BLOCK + tl.arange(0, BLOCK)
    mask = offs < n                                # guard the tail block
    tl.store(out_ptr + offs, tl.load(x_ptr + offs, mask=mask)
             + tl.load(y_ptr + offs, mask=mask), mask=mask)


def add(x, y):
    out = torch.empty_like(x)
    n = out.numel()
    add_kernel[(triton.cdiv(n, 1024),)](x, y, out, n, BLOCK=1024)
    return out
    # Verifier would then run: torch.testing.assert_close(add(x, y), x + y)  # oracle
    # and time add() against x + y on the target GPU before accepting.

The takeaway is that nothing here bypasses the stack: the LLM writes Triton, Triton compiles per architecture, and the verifier checks the result exactly as it would for a human-written kernel.

How to run it in production (verification discipline)¶

This is the load-bearing operational practice. Every accepted artifact must carry its proof: the reference it was checked against, the hardware it was timed on, and the baseline it beat. Never promote an AI-discovered algorithm to production on a benchmark alone; require independent end-to-end validation, exactly the bar that has kept AlphaTensor's results out of cuBLAS.

Approximate tolerances are a trap here: a kernel that is "close" under np.allclose(atol=1e-6) can still be silently wrong, and on a fleet those errors compound. Bind the proof with an exact content hash (a SHA-256 quote anchor) over the reference output on a fixed probe, so the certificate is reproducible on any machine and any later numeric drift (new GPU generation, new cuBLAS) invalidates it instead of passing. This numpy-only block builds such a certificate, and proves it rejects a 1-ULP corruption that a loose tolerance would wave through, plus a reference-drift case. Runnable with system python3:

import hashlib
import numpy as np


def probe_input(seed, m, k, n):
    """A fixed, reproducible probe (identical across machines) for the anchor."""
    rng = np.random.default_rng(seed)
    return (rng.standard_normal((m, k)).astype(np.float64),
            rng.standard_normal((k, n)).astype(np.float64))


def anchor(array):
    """SHA-256 over canonical bytes: the exact quote-anchor for a numeric result."""
    a = np.ascontiguousarray(array, dtype=np.float64)
    h = hashlib.sha256()
    h.update(str(a.shape).encode())
    h.update(a.tobytes())
    return h.hexdigest()


def certify(candidate, hardware, baseline, seed=1234):
    """Bind candidate output to the reference anchor with full provenance."""
    a, b = probe_input(seed, 32, 48, 40)
    ref_anchor = anchor(a @ b)                       # reference oracle on the probe
    return {"correct": anchor(candidate(a, b)) == ref_anchor,
            "reference_anchor": ref_anchor, "hardware": hardware, "baseline": baseline}


def accept(cert):
    """Promotion gate: correct anchor AND recorded hardware provenance."""
    return bool(cert["correct"]) and bool(cert["reference_anchor"]) and bool(cert["hardware"])


a, b = probe_input(1234, 32, 48, 40)

# 1. Happy path: a correct kernel certifies; the anchor is 64 hex chars and reproducible.
cert = certify(lambda x, y: x @ y, hardware="H100", baseline="cuBLAS-12.4")
assert cert["correct"] and accept(cert)
assert len(cert["reference_anchor"]) == 64
assert anchor(a @ b) == cert["reference_anchor"]     # same probe => same anchor anywhere

# 2. Adversarial: a 1-ULP-scale corruption is REJECTED even though allclose calls it close.
def sneaky(x, y):
    c = x @ y
    c[0, 0] = np.nextafter(c[0, 0], np.inf)          # smallest possible perturbation
    return c
assert np.allclose(a @ b, sneaky(a, b), atol=1e-6)   # the bug hides below a loose tolerance
cert_bad = certify(sneaky, hardware="H100", baseline="cuBLAS-12.4")
assert cert_bad["correct"] is False and not accept(cert_bad)

# 3. Adversarial: reference drift (a library upgrade) invalidates the stored certificate.
assert anchor((a @ b) * (1.0 + 1e-12)) != cert["reference_anchor"]

# 4. Adversarial: a certificate with no hardware provenance is not promotable.
untraceable = dict(cert); untraceable["hardware"] = ""
assert not accept(untraceable)

print("anchor OK: len 64 | reproducible | 1-ULP corruption rejected | drift invalidates | provenance required")

Wire AI into operations through your telemetry, not around it. Feed an LLM/agent assistant from the metrics you already collect (Prometheus-style pipelines, DCGM exporters) so it can flag a GPU-memory drop as a likely leak or data stall, surface ECC-error clustering as an impending HBM fault, or catch a diverging loss early (see observability/monitoring, reliability/RAS). Control problems often have many tunable knobs: power/frequency (power/thermal tuning), cache eviction, congestion control, tensor swap to CPU/NVMe. For these, RL agents can learn workload-adaptive policies that static heuristics (plain LRU, for example) miss.

How to maintain and scale it¶

Maintain. Re-verify on each GPU generation and each compiler/library upgrade: a kernel that won on one architecture or cuBLAS version may not on the next, and the certificate above is designed to flag exactly that drift. Keep a human review gate on the verifier's residual failure band (the 4% of KernelBench Level-2 that was not perfectly correct is where humans belong). Store each accepted artifact with its anchor, hardware tag, and baseline, so an audit can reproduce the acceptance decision.

Scale. The economics scale in two directions. Horizontally, the same generate/verify/refine loop fans out across many kernels and many GPUs in parallel, because each problem is independent (embarrassingly parallel search). The generation budget (NVIDIA's roughly 15 min/problem) times the number of target kernels sets the compute bill, and the RL path (GRPO over roughly 5,000 steps) is a one-time training cost amortized over every kernel the tuned model later writes. Vertically, the operational layer scales to the fleet: an AI scheduler colocating compute-heavy and bandwidth-heavy jobs, or flagging ECC errors before a long run crashes, raises goodput across datacentre-scale clusters and toward very large model training. Across both, the invariant is the same: throughput of accepted artifacts is bounded by verifier trust, so scaling the generator without scaling verification just scales the risk. See autonomous experimentation loops and agentic RL for the loop generalized beyond kernels.

Failure modes¶

No reliable oracle. The loop is only as safe as its verifier; without an exact correctness reference, a fast-but-wrong kernel gets accepted. Build the oracle first (KernelBench-style), gate on it, and keep a human on the residual band, since Level-2 correctness was 96%, not 100% (NVIDIA Technical Blog).
Loose-tolerance acceptance. Checking "close enough" (allclose with a generous atol) lets silent numeric bugs through; the SHA-anchor block above shows a 1-ULP error passing a loose tolerance but failing the exact check. Use an exact oracle where the operation is exact.
Hardware-specific wins assumed portable. AlphaTensor's speedup was V100-specific and cuBLAS-version-specific (Fregly, Ch. 20). A kernel or decomposition that won on one architecture can lose on the next; re-verify per GPU generation and per library upgrade.
Promoting on a benchmark alone. As of the book's writing, AlphaTensor's algorithms remained experimental and were not merged into cuBLAS pending validation (Fregly, Ch. 20). Treat AI-discovered algorithms as candidates; require independent end-to-end validation before production.
Reward hacking / zero-signal groups. In the RL path, a reward that credits speed without a hard correctness gate is gamed by fast wrong kernels; and a group where every sample is identical yields zero advantage (no learning), which must be handled (the epsilon guard above) rather than crashing on divide-by-zero.
Runaway generation cost. Inference-time scaling (roughly 15 min/problem) and RL training (roughly 5,000 steps) are real compute bills; without an iteration cap and generation budget the search burns GPU-hours with diminishing returns.
Over-trusting forward-looking autonomy. Fully self-improving agents, perpetual continual learning, and fully automatic pipeline parallelism are research/projection in the book, not shipping capability (Fregly, Ch. 20). Architecting production around them is a date-sensitive mistake.

References¶

Chris Fregly, AI Systems Performance Engineering (O'Reilly), Chapter 20: "AI-Assisted Performance Optimizations and Scaling Toward Multimillion GPU Clusters."
NVIDIA Technical Blog: Automating GPU Kernel Generation with DeepSeek-R1 and Inference Time Scaling (H100; roughly 15 min/problem; 1.1x to 2.1x over FlexAttention; KernelBench Level-1 100% / Level-2 96%).
DeepMind: Discovering faster matrix multiplication algorithms with reinforcement learning, Nature 610 (2022) (AlphaTensor; V100 10 to 20% over cuBLAS-era baseline); project blog; code.
Predibase: Reinforcement Fine-Tuning for LLMs (GRPO; Qwen2.5-Coder-32B-Instruct; KernelBench; Triton). Per-experiment figures (roughly 40% success after around 5,000 steps, up to 3x) as reported in Fregly, Ch. 20.
Stanford: KernelBench (correctness/speed benchmark used as the verifier oracle).
Merouani, Kara Bernou, Baghdadi: Agentic Auto-Scheduling: LLM-Guided Loop Optimization (ComPilot; PACT 2025; PolyBench geomean 2.66x single / 3.54x best-of-5; competitive with the Pluto polyhedral optimizer, zero-shot).