Markdown

Serving and RL post-training on AMD GPUs (ROCm)¶

Scope: how to serve, and reinforcement-learning post-train, known open-weight models on AMD Instinct GPUs via ROCm: the software stack (HIP, RCCL, Composable Kernel, AITER), the engines that run on it (vLLM, SGLang), the quantization toolkit (AMD Quark), and the central portability question this page exists to answer: does training a model on NVIDIA hardware affect whether it can be served, and RL post-trained, on AMD. This is the vendor-portability layer underneath the engine-focused pages inference serving and disaggregated inference.

As of this writing, serving known open-weight models on AMD Instinct GPUs via vLLM or SGLang is production-deployed, not experimental: real deployments are cited below, including bare-metal OCI MI300X and llm-d prefill/decode disaggregation over RoCE. A model's weights, trained on NVIDIA, port to AMD without retraining. What does not port automatically is the kernels, the quantization format and its kernel coverage on the target silicon, and the attention-backend choice; each needs its own validation pass. Calibration is not on that list, and an earlier version of this page wrongly said it was: see the portability section below. Treat a cross-vendor move the way you would treat moving to a new GPU generation within one vendor: the math is the same, the fastest path to it is not, and a few bleeding-edge features are still ahead of their ROCm port.

What it is¶

ROCm is AMD's CUDA-equivalent software stack: HIP is the CUDA-compatible API layer (AMD's hipify tool converts CUDA source to HIP with mostly-automatic translation), RCCL is the NCCL-API-compatible collective communication library, hipBLASLt/Composable Kernel (CK) are AMD's GEMM and fused-kernel template libraries (CK is architecturally similar to NVIDIA's CUTLASS), and AITER (AMD's curated high-performance inference kernel library: fused MoE, attention, GEMM) is the layer most comparable to DeepEP/FlashAttention on the NVIDIA side. Triton is the same compiler used on NVIDIA; it also targets AMD.
Instinct hardware: MI200s (gfx90a), MI300X (gfx942), MI350X/MI355X (gfx950), plus Radeon RX 7900/9000 and Ryzen AI MAX consumer/APU parts.
Engines: vLLM has native ROCm support (ROCm 6.3+, prebuilt wheels for ROCm 7.0 and 7.2.1), and SGLang and Hugging Face TGI also ship ROCm backends. TensorRT-LLM does not: it is an NVIDIA-only compiled-engine format, not a portable artifact (see "When to use it," below).
Quantization: AMD Quark is AMD's quantization toolkit, producing weight/activation/KV-cache quantized checkpoints (FP8 per-tensor and per-channel; MXFP4/MXFP6, OCP Open Compute Project-compliant) via its own calibration workflow (load, prepare calibration data, configure, quantize and export, evaluate), using algorithms including AWQ, GPTQ, Rotation, and AutoSmoothQuant. AMD republishes Quark-quantized variants of well-known open models under the amd/ Hugging Face org, for example amd/Llama-3.3-70B-Instruct-FP8-KV, amd/gpt-oss-20b-w-mxfp4-a-bf16, amd/GLM-4.7-MXFP4, and amd/MiniMax-M2.1-MXFP4.
KV-cache transfer / disaggregation: RIXL is the ROCm analogue of NIXL, built on the UCX transport, used by llm-d to run prefill/decode disaggregation on AMD hardware.

Why use it¶

Avoiding single-vendor capacity constraints. AMD Instinct capacity is a real, separately-sourced alternative when NVIDIA allocations are the bottleneck, and this is not theoretical: llm-d documents a validated deployment on two 8xMI300X nodes over RoCE, and a production-shaped Helm configuration serves amd/Llama-3.3-70B-Instruct-FP8-KV disaggregated (TP=4 decode, four TP=1 prefill replicas) on Oracle Cloud's bare-metal BM.GPU.MI300X.8 shape.¹⁴¹⁵
The ecosystem is real and actively maintained, not a side project. AMD-sponsored ROCm support has landed in vLLM, SGLang, TGI, verl, slime, and the vLLM Semantic Router, and independent academic benchmarking now spans multiple accelerator vendors and frameworks side by side, not just AMD's own numbers: a comprehensive inference-benchmarking suite evaluates LLaMA, Mistral, and Qwen models across NVIDIA and AMD GPUs plus Intel Habana and SambaNova accelerators, on top of MI300X-specific studies of compute throughput, memory bandwidth, and interconnect characterization, and full MoE pretraining at scale.²³⁵
Vendor-neutral benchmark parity is not established by this page. This research did not manage to pull exact MI300X/MI325X versus H100/H200 throughput figures from MLCommons' interactive results dashboard (JavaScript-rendered, not fetchable as static content), so do not treat hardware-parity claims as verified here. Use MLPerf only after pulling the exact submission rows you need, and still run a local validation pass for your model, engine, precision, and concurrency target.

Does training on NVIDIA affect serving (and RL) on AMD?¶

The weights are hardware-agnostic; the surrounding stack is not. A checkpoint is a tensor format (safetensors, a PyTorch state dict): a BF16/FP16 model trained on NVIDIA loads on AMD via vLLM-ROCm or SGLang-ROCm exactly as it would load on a different NVIDIA generation, no conversion step. What does not transfer automatically is everything the serving stack does with those weights: the kernels, the quantization format and whether the target silicon has kernels for it, and the attention-backend choice.

Cross-vendor numerical differences are real, measured, and not a sign of brokenness. Lawrence Livermore National Laboratory's differential-testing study generated over 600,000 short numerical test cases in CUDA and their HIP translations and found "subtle numerical differences that come from (1) math library calls, (2) differences in floating [point behavior]" between NVIDIA and AMD platforms.¹ This is the same category of difference you would see across two CUDA/cuBLAS versions on the same vendor: floating-point operations are not associative, and a different kernel implementation can legitimately produce a slightly different rounding of the same computation. The correct response is the one this KB already recommends for any kernel or version change: validate with output or eval-metric comparison, not a bit-exact diff.

Quantized checkpoints are the real portability boundary, and the boundary is the format and its kernels, not the calibration. An earlier version of this page said an NVIDIA-calibrated checkpoint needs vendor-specific recalibration. That was wrong, and the correction matters because it changes what you actually have to do. Calibration measures the model's activation distribution on your data: it is a property of the model, the recipe, and the calibration set, not of the silicon that ran the forward passes. AMD's own toolkit settles the point, since Quark's PyTorch flow "supports ROCm and CUDA accelerators" and AMD describes it as enabling deployment "across various hardware platforms."⁶ What is genuinely vendor-bound is narrower and more concrete:

Format packing. NVFP4 is NVIDIA Blackwell's microscaling format and has no Instinct kernels; that checkpoint really does not port. OCP FP8 and MXFP4/MXFP6, by contrast, are open standards and are not AMD-only. Blackwell consumes MXFP4 too.
The FP8 variant, which is a conversion and not a wall. MI300-class silicon (gfx942) uses the FNUZ FP8 variant, so an e4m3fn checkpoint produced on NVIDIA is converted at load rather than rejected: vLLM's normalize_e4m3fn_to_e4m3fnuz rewrites the -128 bit pattern (zero in e4m3fn, NaN in e4m3fnuz) and doubles the weight and input scales, because "for the same bits representation, e4m3fnuz value is half of the e4m3fn value."⁷ The calibrated scales survive that conversion by construction. This is the concrete refutation of "NVIDIA FP8 does not port to AMD," and it is worth knowing before you spend a week recalibrating something you could have loaded.
Kernel coverage per format, architecture, and granularity. This is where the real risk sits: an unsupported format or granularity can fall back to slow kernels without erroring, and per-model-family attention support still varies (see the architecture-aware study below).

So why does AMD republish its own quantized variants (amd/Llama-3.3-70B-Instruct-FP8-KV, amd/gpt-oss-20b-w-mxfp4-a-bf16, amd/GLM-4.7-MXFP4)? Because they are pre-validated against the ROCm kernel path AMD supports, which is a convenience and support argument, not evidence that calibration is vendor-bound. Prefer them when they exist. Do not conclude that the NVIDIA-produced checkpoint is unusable.

Recalibration is still sometimes right, for reasons that have nothing to do with the vendor. Recalibrate when the published checkpoint's calibration set does not resemble your traffic, or when you re-target a different format or granularity (per-tensor to per-channel, FP8 to MXFP4). The executed model below quantifies that cost, and it is worth being exact about what it does and does not show.

Architecture-specific kernel gaps are real and must be checked per model family. A cross-architecture study on 8-GPU MI325X clusters running vLLM v0.14.1, spanning four models from 235B to one trillion parameters across MoE+MLA, Dense+GQA, and MoE+GQA families, found that "architecture-aware optimization is essential: MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both," that "the AMD AITER runtime is required for competitive MLA inference throughput and must be selectively disabled for architectures with incompatible attention head configurations," and that a controlled AITER ablation on Llama-3.1-405B "reveals a modest 3-5% throughput benefit at high concurrency but 2-16x higher measurement variability, confirming that AITER's large speedups target MoE/MLA kernels specifically."⁴ This is exactly why vLLM's ROCm attention-backend selector exists as an explicit choice (ROCM_ATTN, TRITON_ATTN, TRITON_MLA, ROCM_AITER_MLA) rather than one default: the right backend depends on the model's attention architecture, and picking the wrong one silently costs throughput or breaks a memory-offload path.⁸

Full RL post-training on AMD is demonstrated, not hypothetical. verl publishes reproducible RL baselines, PPO, GRPO, ReMax, SPPO, RLOO, and SPIN, across models from Gemma to Qwen and Mixtral, run on both NVIDIA GPU and AMD MI300 hardware against GSM8k, with linked command logs and Weights & Biases runs for reproducibility.¹⁰ Getting there requires a ROCm-specific build: a ROCm/NUMA-patched base image with SGLang and vLLM built from source for custom gfx90a/gfx942 architecture targets, and careful HIP_VISIBLE_DEVICES handling under Ray via the RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES/RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES flags.¹¹ Current rough edges are documented, not hidden: vLLM's sleep-mode memory offload, needed to fit long-context or multi-node rollout, has landed in vLLM's main branch (vLLM > 0.11.0) on ROCm >= 7.0, and multi-node CUDA-graph capture under vLLM's V1 engine needs its capture sizes explicitly capped (cudagraph_capture_sizes) with eager mode off (enforce_eager=False).¹² slime independently hit and fixed a ROCm/HIP-specific systems bug: setting non_blocking=True during a tensor transfer, harmless on CUDA, pins the tensor's memory on ROCm/HIP and segfaults when a subprocess later forks; the fix forces non_blocking=False whenever torch.version.hip is detected.¹³ At the far end of the same spectrum, a full-scale mixture-of-experts pretraining run has been completed on pure AMD hardware (MI300X compute with Pollara networking), which is stronger evidence than any inference-only benchmark that the entire training stack, not just serving, works end to end on AMD; RL post-training is a lighter-weight version of that same infrastructure pattern.⁵

Net verdict. Portability is real and already in production for both serving and RL, but "just works" is the wrong mental model. A cross-vendor move needs the same validation discipline as a cross-generation move within one vendor (numerics, quantization format and kernel coverage, backend selection), and a handful of bleeding-edge features are simply not there yet: this KB's own DSpark speculative decoding page documents that vLLM merged native CUDA support on 2026-07-01 while "ROCm support is still an open draft PR, and hardware-specific bugs were still being filed against it as of 2026-07-05."¹⁷

When to use it (and when not)¶

Use AMD/ROCm serving when capacity or economics favor it, the model's attention architecture has a confirmed, validated backend (dense GQA models are the safer starting point per the architecture-aware study above; MoE+MLA models need explicit AITER validation, not a default assumption), and a validation pass is budgeted before shipping.
Use AMD/ROCm for RL post-training when a verl (or SGLang-rollout) ROCm-built image is acceptable and its version floors (ROCm >= 7.0, vLLM > 0.11.0) and the multi-node CUDA-graph capture-size cap are tolerable, rather than requiring a fully hands-off path.
Be cautious, or wait, when the technique is bleeding-edge. Verify ROCm support per feature rather than assuming parity by default; DSpark speculative decoding (still an open ROCm draft PR while CUDA support is merged) is a concrete, already-documented example in this KB.¹⁷
Do not assume a compiled, vendor-specific artifact is portable. A TensorRT-LLM engine or an NVFP4 (Blackwell-specific microscaling) checkpoint is compiled or packed for one vendor's silicon. Only the underlying open checkpoint, the safetensors/Hugging Face format, is the portable artifact; a compiled engine built from it is not.

Architecture¶

One artifact crosses the vendor boundary unchanged (the open checkpoint) and one does not (a packing or engine format compiled for specific silicon). Everything in between is a format-and-kernel question. Calibration is not on the boundary at all: it belongs to the model and the data, and it travels with the checkpoint.

flowchart TB
  W["Checkpoint trained on NVIDIA<br/>(safetensors / state dict, BF16 or FP16)"]
  W -->|"loads unchanged"| N["Served on NVIDIA<br/>CUDA, NCCL, cuBLAS/CUTLASS, FlashAttention, TensorRT-LLM"]
  W -->|"loads unchanged, validate numerics"| A["Served on AMD<br/>HIP, RCCL, hipBLASLt/Composable Kernel, AITER, vLLM-ROCm/SGLang-ROCm"]
  W -->|"quantize (Quark or ModelOpt;<br/>either toolkit runs on either vendor's GPU)"| Q["Quantized checkpoint:<br/>calibration is model + data + recipe,<br/>NOT silicon"]
  Q -->|"OCP FP8 e4m3fn"| A
  Q -->|"OCP MXFP4 / MXFP6"| A
  Q -->|"OCP FP8 / MXFP4"| N
  Q -->|"NVFP4 (Blackwell microscaling packing)"| N
  Q -.->|"NVFP4: no Instinct kernels,<br/>this is the real non-portable case"| X["Blocked on AMD"]
  A -.- F["gfx942 FP8 is the FNUZ variant:<br/>vLLM converts e4m3fn at load<br/>(-128 bit pattern zeroed, scales x2).<br/>A conversion, not a wall."]

How to use it: engine install and backend selection¶

vLLM's ROCm wheels carry a documented trap worth calling out on its own: pre-built ROCm wheels are only available for Python 3.12. On any other Python version the installer silently falls back to the CUDA wheel from PyPI, which then fails on AMD hardware with an unrelated-looking error (libcudart.so: cannot open shared object file) rather than a clear version mismatch.¹⁶

# Check Python version first; ROCm wheels require exactly 3.12.
python3 --version

# If needed, an isolated 3.12 environment:
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate

# Install vLLM for ROCm 7.0 (adjust the extra index for your ROCm version).
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade

The attention backend is not automatic; vLLM's ROCm selector picks an implementation depending on the model's architecture, block size, and an explicit AITER flag. An explicitly requested backend is validated against its own constraints and either used exactly as requested or rejected outright, there is no silent substitution to a different backend:⁸

Requested backend	`use_mla`	`block_size`	Outcome
`ROCM_ATTN`	`False`	unset or a multiple of 16	`ROCM_ATTN`
`ROCM_ATTN`	`False`	`1`	raises `ValueError` (block size not supported)
`ROCM_ATTN`	`True`	any	raises `ValueError` (MLA not supported)
`TRITON_MLA`	`True`	unset or a multiple of 16	`TRITON_MLA`
`TRITON_MLA`	`True`	`1`	raises `ValueError` (block size not supported)
`ROCM_AITER_MLA`	`True`	any, including `1`	`ROCM_AITER_MLA` (block-size-agnostic; explicit requests bypass `VLLM_ROCM_USE_AITER` entirely)
unspecified	`True`	unset or a multiple of 16	`ROCM_AITER_MLA` if AITER-MLA is enabled (`VLLM_ROCM_USE_AITER=1`, since `VLLM_ROCM_USE_AITER_MLA` defaults on); otherwise `TRITON_MLA`
unspecified	`True`	`1`	`ROCM_AITER_MLA` if AITER-MLA is enabled; otherwise raises `ValueError` (no valid backend)
unspecified	`False`	unset or a multiple of 16	`ROCM_ATTN`, or `ROCM_AITER_UNIFIED_ATTN`/`ROCM_AITER_FA` if `ROCM_ATTN` is excluded (for example, a KV connector is active)

A real MoE serving configuration wires this together with Quark's MXFP4 output and AMD's MoE-specific AITER kernels, for example an evaluation config serving amd/gpt-oss-20b-w-mxfp4-a-bf16:⁹

# Serve an AMD Quark MXFP4/BF16 checkpoint on ROCm with AITER-accelerated attention and MoE.
export VLLM_ROCM_USE_AITER=1
vllm serve amd/gpt-oss-20b-w-mxfp4-a-bf16 \
  --tensor-parallel-size 2 \
  --attention-backend ROCM_AITER_UNIFIED_ATTN \
  --moe-backend aiter

What a calibration mismatch costs, and what this does not prove. The model below is a controlled illustration, not a bit-exact FP8 encoder and not a reproduction of anyone's real calibration data. It quantizes an activation distribution using a clip range fitted to that same data, versus a clip range borrowed from a different distribution, and shows the borrowed range costs real accuracy.

Read the scope precisely, because the earlier version of this page did not. The effect it demonstrates is a calibration-data mismatch, which is a distribution-shift effect: it is exactly as large when both runs happen on the same vendor's GPU, and it says nothing about NVIDIA versus AMD. The two distributions here differ by construction, so the model cannot and does not establish a vendor effect. It is the right argument for "recalibrate when the published calibration set does not match your traffic" and the wrong argument for "recalibrate because you changed GPU vendor."

# quant_calibration_mismatch.py -- validated: does reusing one calibration's clip range
# for FP8 quantization on a differently-scaled activation distribution cost accuracy,
# versus recalibrating on the actual data? Run: python3 quant_calibration_mismatch.py
from __future__ import annotations

import numpy as np


def fp8_e4m3_quantize(x: np.ndarray, clip: float) -> np.ndarray:
    """Simulate symmetric FP8 E4M3-like quantization: clip to a calibrated range,
    then quantize to 2**3 = 8 mantissa steps per octave (illustrative, not a bit-exact
    E4M3 encoder). Captures the two real costs of calibration: clipping and step size."""
    x_clipped = np.clip(x, -clip, clip)
    step = clip / 448.0  # 448 is the E4M3 max normal magnitude; sets the quantization grid
    levels = 8  # illustrative mantissa resolution, not a full E4M3 bit-layout simulation
    q = np.round(x_clipped / step * levels) / levels * step
    return q


def quant_error(x: np.ndarray, clip: float) -> float:
    return float(np.mean((x - fp8_e4m3_quantize(x, clip)) ** 2))


if __name__ == "__main__":
    rng = np.random.default_rng(0)

    # Two activation distributions with the SAME shape family but different scale/outlier
    # behavior: "the calibration set the published checkpoint was fitted on" vs "the
    # activations this weight actually sees on YOUR traffic." NOTE the framing: this is a
    # DATA mismatch, not a vendor mismatch. Nothing here is vendor-specific, and running
    # both sides on the same GPU would produce the identical penalty.
    # Real transformer activations are heavy-tailed; a Student-t stand-in is illustrative.
    calib_data = rng.standard_t(df=3, size=200_000) * 1.0      # published calibration set
    served_data = rng.standard_t(df=3, size=200_000) * 1.6     # your traffic, 60% larger scale

    # Recalibrated: clip range derived from the SAME distribution being quantized.
    clip_matched = float(np.percentile(np.abs(served_data), 99.9))
    err_matched = quant_error(served_data, clip_matched)

    # Borrowed: clip range fitted to the OTHER distribution, then applied as-is.
    clip_borrowed = float(np.percentile(np.abs(calib_data), 99.9))
    err_borrowed = quant_error(served_data, clip_borrowed)

    assert clip_borrowed < clip_matched, "the borrowed range must under-cover the wider real data"
    assert err_borrowed > err_matched, "a mismatched calibration range must cost accuracy"

    penalty = (err_borrowed - err_matched) / err_matched
    assert penalty > 0.05, f"the mismatch penalty should be clearly non-trivial, got {penalty:.3f}"

    # Sanity: quantizing with a clip range that matches the true distribution should always
    # dominate a range calibrated on a materially different distribution, at any tested scale gap.
    for scale_gap in (1.1, 1.6, 2.5):
        served = rng.standard_t(df=3, size=50_000) * scale_gap
        matched = quant_error(served, float(np.percentile(np.abs(served), 99.9)))
        borrowed = quant_error(served, float(np.percentile(np.abs(calib_data), 99.9)))
        assert matched <= borrowed, (scale_gap, matched, borrowed)

    print(f"clip (recalibrated on the served data) = {clip_matched:.3f}")
    print(f"clip (borrowed from a mismatched calib set) = {clip_borrowed:.3f}")
    print(f"quant MSE, recalibrated = {err_matched:.6f}")
    print(f"quant MSE, borrowed clip = {err_borrowed:.6f}")
    print(f"borrowed-clip error penalty = {penalty*100:.1f}% worse than recalibrating")
    print("NOTE: this is a data-distribution effect, not a vendor effect.")
    print("ALL ASSERTIONS PASSED")

Executed output:

clip (recalibrated on the served data) = 21.402
clip (borrowed from a mismatched calib set) = 12.794
quant MSE, recalibrated = 0.240987
quant MSE, borrowed clip = 0.530887
borrowed-clip error penalty = 120.3% worse than recalibrating
NOTE: this is a data-distribution effect, not a vendor effect.
ALL ASSERTIONS PASSED

The mechanism, not the exact percentage, is the point: a clip range calibrated on the wrong distribution both under-covers the real dynamic range (clipping outliers) and wastes quantization steps on a narrower range than the data actually spans. That is what any calibration pass, Quark's or ModelOpt's, exists to avoid, and it is an argument about data, not about silicon. The vendor-facing conclusion it supports is modest and worth stating in full: prefer an amd/-published Quark checkpoint on ROCm because it is pre-validated against the kernel path AMD supports, and recalibrate when your traffic does not look like the published calibration set. It is not a reason to treat an NVIDIA-produced OCP FP8 checkpoint as unusable on Instinct, because vLLM loads exactly that, converting e4m3fn to the gfx942 FNUZ variant on the way in.⁷

How to develop with it: RL post-training on AMD¶

verl's ROCm path is a distinct build, not a flag on the standard image: a ROCm/NUMA-patched Ubuntu base image with SGLang and vLLM built from source for custom gfx90a/gfx942 architecture targets, and both single-node (PPO/GRPO) and multi-node SLURM-orchestrated training supported, with vLLM as the default rollout engine and SGLang also supported.¹¹ Ray's device-visibility handling is the sharp edge: launching Ray needs HIP_VISIBLE_DEVICES set alongside Ray's own RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES/RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES flags, so pin and verify against the current verl tutorial rather than assuming CUDA-style device-visibility handling carries over.¹¹

Two concrete, already-documented rough edges to plan around:

vLLM sleep mode has landed on ROCm, but pin the version floor. It offloads unused GPU memory to CPU between rollout and training phases, which matters for long-context and multi-node RL; it is merged into vLLM's main branch (vLLM > 0.11.0) and needs ROCm >= 7.0, no community-patched branch required.¹²
Cap CUDA-graph capture sizes for multi-node ROCm deployments. Set cudagraph_capture_sizes explicitly and keep enforce_eager=False; re-check this guidance against the current verl tutorial before every vLLM bump, since it has already changed once.¹²

slime's fix for the ROCm/HIP pinned-memory segfault is a useful template for the general class of bug to expect: detect the platform (torch.version.hip), and override the one CUDA-safe default that is not HIP-safe, rather than patching every call site.¹³

How to maintain it¶

Do not assume AITER is a blanket win; verify it per architecture. The controlled ablation above found a 3-5% benefit at high concurrency but 2-16x higher run-to-run variability on a dense model, "confirming that AITER's large speedups target MoE/MLA kernels specifically."⁴ Measure before and after enabling it on your actual model family.
Track vendor-published quantized checkpoints rather than hand-rolling quantization. AMD's amd/ org and Quark's calibration workflow are the supported path for FP8/MXFP4 on ROCm; re-run Quark's own calibration if you must quantize a model AMD has not already published.
Re-validate after every ROCm, vLLM, or SGLang bump, with the same discipline this KB already applies to driver/CUDA upgrades on NVIDIA: numerics (output or eval-metric comparison, not bit-exact diffing), attention-backend selection, and memory/graph-capture behavior are all points that have moved between releases in the sources cited on this page.

Running it in production¶

	What is production-composable today	What is still a rough edge
Serving	vLLM/SGLang ROCm backends; llm-d P/D disaggregation with RIXL over RoCE, validated on real hardware; Quark-quantized `amd/`-published checkpoints for known models¹⁴¹⁵	Attention-backend and offload choices are architecture-dependent, not automatic; verify per model family⁴
RL post-training	verl reproducible cross-hardware baselines (NVIDIA GPU and AMD MI300) on real RL algorithms (PPO, GRPO, ReMax, SPPO, RLOO, SPIN)¹⁰; vLLM sleep mode merged into main on ROCm >= 7.0¹²	Multi-node CUDA-graph capture needs an explicit capture-size cap and `enforce_eager=False`¹²
Pretraining	Full-scale MoE pretraining demonstrated on pure AMD hardware (MI300X plus Pollara networking)⁵	Ecosystem is younger than NVIDIA's; expect to write the networking/collective characterization yourself for a novel cluster shape
Bleeding-edge techniques	N/A	Verify per feature; some (DSpark speculative decoding) are CUDA-merged but ROCm-draft-PR-only as of this KB's last check¹⁷

Failure modes¶

Silent Python-version wheel fallback. Installing vLLM's ROCm wheel on anything other than Python 3.12 silently pulls the CUDA wheel instead, which then fails on AMD hardware with a confusing libcudart.so error rather than a clear version mismatch.¹⁶
Assuming AITER is a uniform win. Applying it to an attention head configuration it does not support, or expecting the same speedup magnitude on a dense model that a MoE/MLA model would see.⁴
Reusing an NVIDIA-targeted FP8/MXFP4 checkpoint's calibration constants directly instead of an AMD Quark-recalibrated amd/-published variant; the executed model above quantifies the accuracy this costs.
CUDA-graph capture crashing multi-node ROCm vLLM V1 deployments without an explicit cudagraph_capture_sizes cap and enforce_eager=False.¹²
Leaving non_blocking=True at its CUDA-safe default in a training or checkpointing path on ROCm/HIP, which pins memory and segfaults on a later subprocess fork.¹³
Assuming a compiled, vendor-specific artifact is portable. A TensorRT-LLM engine or an NVFP4 checkpoint is compiled or packed for one vendor's silicon; only the underlying open checkpoint is.

Open questions & validation¶

Independently reproduce a specific MLPerf-style throughput comparison on your own MI300X/MI325X versus H100/H200 allocation; exact figures were not independently re-derived from the primary MLCommons results dashboard in this research (it is JavaScript-rendered and was not fetchable as static content here), so treat parity claims as unverified until you pull the exact submission rows.
Re-check verl's current ROCm tutorial for its vLLM and ROCm version floors and CUDA-graph capture guidance before every upgrade; this page's own citations already moved once (an older revision required a community-patched vLLM branch and a different CUDA-graph workaround than the current tutorial documents).
Run your own numerical-parity check (torch.allclose or an eval-metric delta, not a bit-exact diff) for the specific model you plan to move from NVIDIA to AMD serving; the LLNL study establishes that cross-vendor differences exist and are measurable in general, not what they are for your specific model and kernel path.
Whether DSpark speculative decoding, or any other bleeding-edge technique you depend on, has closed its ROCm gap since this page's last update; verify per feature rather than assuming parity.

References¶

vLLM documentation, GPU installation (ROCm section): https://docs.vllm.ai/en/latest/getting_started/installation/gpu/
AMD Quark quantization toolkit (vLLM integration docs): https://docs.vllm.ai/en/v0.18.0/features/quantization/quark/
AMD Quark, basic usage and supported platforms (the PyTorch flow supports ROCm and CUDA accelerators): https://quark.docs.amd.com/release-0.11/basic_usage.html
vLLM, ROCm FP8 handling (normalize_e4m3fn_to_e4m3fnuz in vllm/model_executor/layers/quantization/utils/w8a8_utils.py), the load-time conversion that makes an NVIDIA-produced OCP FP8 checkpoint usable on gfx942: https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/utils/w8a8_utils.py
verl (RL post-training library), AMD/ROCm tutorial: https://github.com/verl-project/verl/tree/main/docs/amd_tutorial
verl, RL training baselines on GSM8k across NVIDIA GPU and AMD MI300: https://github.com/verl-project/verl/blob/main/docs/algo/baseline.md
slime (RL library), ROCm/HIP checkpoint-writer compatibility fix: https://github.com/THUDM/slime
llm-d, AMD GPU P/D disaggregation with RIXL: https://github.com/llm-d/llm-d/tree/main/guides/pd-disaggregation
Zahid, Laguna, and Le (LLNL), Testing GPU Numerics: Finding Numerical Differences Between NVIDIA and AMD GPUs (arXiv 2410.09172): https://arxiv.org/abs/2410.09172
LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators (arXiv 2411.00136): https://arxiv.org/abs/2411.00136
AMD MI300X GPU Performance Analysis (arXiv 2510.27583): https://arxiv.org/abs/2510.27583
Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study (arXiv 2603.10031): https://arxiv.org/abs/2603.10031
Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design (arXiv 2511.17127): https://arxiv.org/abs/2511.17127
Chopper: A Multi-Level GPU Characterization Tool and Derived Insights Into LLM Training Inefficiency (arXiv 2512.08242): https://arxiv.org/abs/2512.08242
MLCommons, MLPerf Inference benchmark results (datacenter): https://mlcommons.org/benchmarks/inference-datacenter/

Zahid, Laguna, and Le, arXiv 2410.09172, abstract: "We present a study of compiler-induced numerical differences between NVIDIA and AMD GPUs... We generated more than 600,000 tests and found subtle numerical differences that come from (1) math library calls, (2) differences in floating [point behavior]." ↩
LLM-Inference-Bench, arXiv 2411.00136, abstract: "We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters." ↩
AMD MI300X GPU Performance Analysis, arXiv 2510.27583, abstract: "AMD's latest MI300X GPUs offer a compelling alternative, featuring high HBM capacity, matrix cores, and their proprietary interconnect... a comprehensive evaluation of the AMD MI300X GPUs across key performance domains critical to LLM inference including compute throughput, memory bandwidth, and interconnect communication." ↩
Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs, arXiv 2603.10031, abstract: cross-architecture evaluation on 8-GPU MI325X clusters, vLLM v0.14.1, four models 235B-1T params (MoE+MLA, Dense+GQA, MoE+GQA); "MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both"; "the AMD AITER runtime is required for competitive MLA inference throughput and must be selectively disabled for architectures with incompatible attention head configurations"; controlled AITER ablation on Llama-3.1-405B (n=5 per condition) "reveals a modest 3-5% throughput benefit at high concurrency but 2-16x higher measurement variability, confirming that AITER's large speedups target MoE/MLA kernels specifically." ↩↩↩↩
Training Foundation Models on a Full-Stack AMD Platform, arXiv 2511.17127, abstract: "the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking"; cluster/networking characterization for core collectives "to our knowledge, the first at this scale." ↩↩↩
AMD Quark documentation, basic usage: the PyTorch flow "supports ROCm and CUDA accelerators," and AMD describes Quark as enabling "efficient deployment across various hardware platforms." Quark is therefore not evidence that calibration is vendor-bound; it is a toolkit that runs on either vendor's GPU. https://quark.docs.amd.com/release-0.11/basic_usage.html ↩
vLLM, normalize_e4m3fn_to_e4m3fnuz (vllm/model_executor/layers/quantization/utils/w8a8_utils.py): MI300-class gfx942 uses the FNUZ FP8 variant, so vLLM converts an e4m3fn checkpoint at load. The code zeroes the -128 bit pattern ("The bits pattern 10000000(-128) represents zero in e4m3fn but NaN in e4m3fnuz. So here we set it to 0.") and doubles weight_scale and input_scale ("For the same bits representation, e4m3fnuz value is half of the e4m3fn value, so we should double the scaling factor to get the same dequantized value."). The calibrated scales are preserved through the conversion, which is why an NVIDIA-produced OCP FP8 checkpoint is loadable on Instinct rather than requiring recalibration. Read against main, 2026-07-12. https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/utils/w8a8_utils.py ↩↩
vLLM ROCm attention-backend selection (vllm/platforms/rocm.py, RocmPlatform.get_attn_backend_cls and _get_backend_priorities, dispatched from vllm/v1/attention/selector.py; active test tests/kernels/attention/test_attention_selector.py::test_backend_selection on the main branch): an explicit request is checked only against that backend's own validate_configuration, never substituted for a different backend. TRITON_MLA requires block_size unset or a multiple of 16; ROCM_AITER_MLA accepts any block_size because its kernel always operates at page granularity 1 internally, and VLLM_ROCM_USE_AITER is not consulted for explicit requests, only for auto-selection priority when no backend is named. ↩↩
vLLM evaluation config, tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-aiter.yaml: serves amd/gpt-oss-20b-w-mxfp4-a-bf16 with ROCM_AITER_UNIFIED_ATTN attention backend, aiter MoE backend, VLLM_ROCM_USE_AITER=1, tensor-parallel-size 2. ↩
verl, docs/algo/baseline.md: reproducible RL baselines on GSM8k across models (Gemma, Qwen, DeepSeek, Mixtral), methods (SFT, PPO, GRPO, ReMax, SPPO, RLOO, SPIN), and hardware platforms (NVIDIA GPU, AMD MI300), with linked command logs and Weights & Biases runs. ↩↩
verl, docs/amd_tutorial/amd_build_dockerfile_page.rst: the ROCm build Dockerfile starts FROM a ROCm/NUMA-patched Ubuntu base image (rlsys/rocm-6.3.4-patch:rocm6.3.4-numa-patch_ubuntu-22.04) and builds SGLang and vLLM from source for custom gfx90a/gfx942 architecture targets; Ray launch needs HIP_VISIBLE_DEVICES plus the RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES/RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES flags; supports single-node (PPO/GRPO) and multi-node SLURM training with both vLLM (default) and SGLang as rollout engines. ↩↩↩
verl, docs/amd_tutorial/amd_vllm_page.rst (current revision, merged 2025-11-13 via verl PR #4108, superseding an April-2025 revision that required a community-patched vLLM branch and a different CUDA-graph workaround): vLLM sleep mode is merged into vLLM's main branch for versions later than 0.11.0, recommends ROCm >= 7.0; CUDA-graph capture on multi-node AMD deployments needs actor_rollout_ref.rollout.cudagraph_capture_sizes set explicitly with enforce_eager=False. ↩↩↩↩↩↩
slime, slime/utils/rocm_checkpoint_writer.py: on ROCm/HIP, non_blocking=True during tensor operations causes tensors to be stored in pinned memory, which triggers segmentation faults when subprocesses are forked; ROCmFileSystemWriterAsync detects HIP via torch.version.hip and forces non_blocking=False. ↩↩↩
llm-d, guides/pd-disaggregation/modelserver/amd/vllm/base/patch-decode.yaml and patch-prefill.yaml (current Kustomize layout; the repo migrated its P/D guides from Helm to Kustomize in April 2026, replacing the earlier README.amd.md): serves amd/Llama-3.3-70B-Instruct-FP8-KV with --attention-backend=ROCM_ATTN required when prefill/decode tensor-parallel sizes differ; RIXL uses UCX (UCX_TLS, UCX_IB_GID_INDEX) as its KV-transfer transport. ↩↩
llm-d, guides/pd-disaggregation/modelserver/amd/vllm/oci/kustomization.yaml: single-node deployment on OCI OKE BM.GPU.MI300X.8, one TP=4 decode pod plus four TP=1 prefill pods, ghcr.io/llm-d/llm-d-rocm:v0.7.0 container image (guides/recipes/modelserver/components/images/amd-vllm/kustomization.yaml). ↩↩
vLLM ROCm installation docs (rendered under the GPU installation page; source file docs/getting_started/installation/gpu.rocm.inc.md): "vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1"; "ROCm pre-built wheels are only available for Python 3.12. If you are using a different Python version... the installer will silently fall back to the CUDA wheel from PyPI, which will fail on AMD GPUs with errors like libcudart.so: cannot open shared object file." ↩↩
This KB, DSpark speculative decoding: vLLM merged native dspark speculative-decoding support 2026-07-01; "ROCm support is still an open draft PR, and hardware-specific bugs were still being filed against it as of 2026-07-05." ↩↩↩