Markdown

Serving and RL post-training on AMD GPUs (ROCm)¶

Scope: how to serve, and reinforcement-learning post-train, known open-weight models on AMD Instinct GPUs via ROCm: the software stack (HIP, RCCL, Composable Kernel, AITER), the engines that run on it (vLLM, SGLang), the quantization toolkit (AMD Quark), and the central portability question this page exists to answer: does training a model on NVIDIA hardware affect whether it can be served, and RL post-trained, on AMD. This is the vendor-portability layer underneath the engine-focused pages inference serving and disaggregated inference.

As of this writing, serving known open-weight models on AMD Instinct GPUs via vLLM or SGLang is production-deployed, not experimental: real deployments are cited below, including bare-metal OCI MI300X and llm-d prefill/decode disaggregation over RoCE. A model's weights, trained on NVIDIA, port to AMD without retraining. The kernels, quantization calibration, and attention-backend choice around those weights do not port automatically and need their own validation pass. Treat a cross-vendor move the way you would treat moving to a new GPU generation within one vendor: the math is the same, the fastest path to it is not, and a few bleeding-edge features are still ahead of their ROCm port.

What it is¶

ROCm is AMD's CUDA-equivalent software stack: HIP is the CUDA-compatible API layer (AMD's hipify tool converts CUDA source to HIP with mostly-automatic translation), RCCL is the NCCL-API-compatible collective communication library, hipBLASLt/Composable Kernel (CK) are AMD's GEMM and fused-kernel template libraries (CK is architecturally similar to NVIDIA's CUTLASS), and AITER (AMD's curated high-performance inference kernel library: fused MoE, attention, GEMM) is the layer most comparable to DeepEP/FlashAttention on the NVIDIA side. Triton is the same compiler used on NVIDIA; it also targets AMD.
Instinct hardware: MI200s (gfx90a), MI300X (gfx942), MI350X/MI355X (gfx950), plus Radeon RX 7900/9000 and Ryzen AI MAX consumer/APU parts.
Engines: vLLM has native ROCm support (ROCm 6.3+, prebuilt wheels for ROCm 7.0 and 7.2.1), and SGLang and Hugging Face TGI also ship ROCm backends. TensorRT-LLM does not: it is an NVIDIA-only compiled-engine format, not a portable artifact (see "When to use it," below).
Quantization: AMD Quark is AMD's quantization toolkit, producing weight/activation/KV-cache quantized checkpoints (FP8 per-tensor and per-channel; MXFP4/MXFP6, OCP Open Compute Project-compliant) via its own calibration workflow (load, prepare calibration data, configure, quantize and export, evaluate), using algorithms including AWQ, GPTQ, Rotation, and AutoSmoothQuant. AMD republishes Quark-quantized variants of well-known open models under the amd/ Hugging Face org, for example amd/Llama-3.3-70B-Instruct-FP8-KV, amd/gpt-oss-20b-w-mxfp4-a-bf16, amd/GLM-4.7-MXFP4, and amd/MiniMax-M2.1-MXFP4.
KV-cache transfer / disaggregation: RIXL is the ROCm analogue of NIXL, built on the UCX transport, used by llm-d to run prefill/decode disaggregation on AMD hardware.

Why it matters¶

Avoiding single-vendor capacity constraints. AMD Instinct capacity is a real, separately-sourced alternative when NVIDIA allocations are the bottleneck, and this is not theoretical: llm-d documents a validated deployment on two 8xMI300X nodes over RoCE, and a production-shaped Helm configuration serves amd/Llama-3.3-70B-Instruct-FP8-KV disaggregated (TP=4 decode, four TP=1 prefill replicas) on Oracle Cloud's bare-metal BM.GPU.MI300X.8 shape.¹²¹³
The ecosystem is real and actively maintained, not a side project. AMD-sponsored ROCm support has landed in vLLM, SGLang, TGI, verl, slime, and the vLLM Semantic Router, and independent academic benchmarking now spans multiple accelerator vendors and frameworks side by side, not just AMD's own numbers: a comprehensive inference-benchmarking suite evaluates LLaMA, Mistral, and Qwen models across NVIDIA and AMD GPUs plus Intel Habana and SambaNova accelerators, on top of MI300X-specific studies of compute throughput, memory bandwidth, and interconnect characterization, and full MoE pretraining at scale.²³⁵
Reported hardware parity is directionally credible but was not independently re-derived here. AMD's MI300X and MI325X have been reported in industry press as achieving performance close to NVIDIA's H100 and H200 respectively on MLPerf Inference submissions; this research did not manage to pull the exact throughput figures from MLCommons' own interactive results dashboard (JavaScript-rendered, not fetchable as static content), so treat the parity claim as widely reported, not verified against the primary results table in this page.

Does training on NVIDIA affect serving (and RL) on AMD?¶

The weights are hardware-agnostic; the surrounding stack is not. A checkpoint is a tensor format (safetensors, a PyTorch state dict): a BF16/FP16 model trained on NVIDIA loads on AMD via vLLM-ROCm or SGLang-ROCm exactly as it would load on a different NVIDIA generation, no conversion step. What does not transfer automatically is everything the serving stack does with those weights: the kernels, the quantization calibration, and the attention-backend choice.

Cross-vendor numerical differences are real, measured, and not a sign of brokenness. Lawrence Livermore National Laboratory's differential-testing study generated over 600,000 short numerical test cases in CUDA and their HIP translations and found "subtle numerical differences that come from (1) math library calls, (2) differences in floating [point behavior]" between NVIDIA and AMD platforms.¹ This is the same category of difference you would see across two CUDA/cuBLAS versions on the same vendor: floating-point operations are not associative, and a different kernel implementation can legitimately produce a slightly different rounding of the same computation. The correct response is the one this KB already recommends for any kernel or version change: validate with output or eval-metric comparison, not a bit-exact diff.

Quantized checkpoints are the real portability boundary, not the unquantized weights. AMD's Quark toolkit runs its own calibration workflow rather than reusing a checkpoint calibrated for a different vendor's kernels, which is why AMD republishes its own quantized variants (amd/Llama-3.3-70B-Instruct-FP8-KV, amd/gpt-oss-20b-w-mxfp4-a-bf16, amd/GLM-4.7-MXFP4) instead of pointing vLLM-ROCm at the original vendor's NVIDIA-targeted FP8 checkpoint. The executed model below quantifies why that matters: reusing a clipping range calibrated on a different activation distribution costs real accuracy, it is not a cosmetic difference.

Architecture-specific kernel gaps are real and must be checked per model family. A cross-architecture study on 8-GPU MI325X clusters running vLLM v0.14.1, spanning four models from 235B to one trillion parameters across MoE+MLA, Dense+GQA, and MoE+GQA families, found that "architecture-aware optimization is essential: MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both," that "the AMD AITER runtime is required for competitive MLA inference throughput and must be selectively disabled for architectures with incompatible attention head configurations," and that a controlled AITER ablation on Llama-3.1-405B "reveals a modest 3-5% throughput benefit at high concurrency but 2-16x higher measurement variability, confirming that AITER's large speedups target MoE/MLA kernels specifically."⁴ This is exactly why vLLM's ROCm attention-backend selector exists as an explicit choice (ROCM_ATTN, TRITON_ATTN, TRITON_MLA, ROCM_AITER_MLA) rather than one default: the right backend depends on the model's attention architecture, and picking the wrong one silently costs throughput or breaks a memory-offload path.⁶

Full RL post-training on AMD is demonstrated, not hypothetical. verl publishes reproducible RL baselines, PPO, GRPO, ReMax, SPPO, RLOO, and SPIN, across models from Gemma to Qwen and Mixtral, run on both NVIDIA GPU and AMD MI300 hardware against GSM8k, with linked command logs and Weights & Biases runs for reproducibility.⁸ Getting there requires a ROCm-specific build: a ROCm/NUMA-patched base image with SGLang and vLLM built from source for custom gfx90a/gfx942 architecture targets, and careful HIP_VISIBLE_DEVICES handling under Ray via the RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES/RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES flags.⁹ Current rough edges are documented, not hidden: vLLM's sleep-mode memory offload, needed to fit long-context or multi-node rollout, has landed in vLLM's main branch (vLLM > 0.11.0) on ROCm >= 7.0, and multi-node CUDA-graph capture under vLLM's V1 engine needs its capture sizes explicitly capped (cudagraph_capture_sizes) with eager mode off (enforce_eager=False).¹⁰ slime independently hit and fixed a ROCm/HIP-specific systems bug: setting non_blocking=True during a tensor transfer, harmless on CUDA, pins the tensor's memory on ROCm/HIP and segfaults when a subprocess later forks; the fix forces non_blocking=False whenever torch.version.hip is detected.¹¹ At the far end of the same spectrum, a full-scale mixture-of-experts pretraining run has been completed on pure AMD hardware (MI300X compute with Pollara networking), which is stronger evidence than any inference-only benchmark that the entire training stack, not just serving, works end to end on AMD; RL post-training is a lighter-weight version of that same infrastructure pattern.⁵

Net verdict. Portability is real and already in production for both serving and RL, but "just works" is the wrong mental model. A cross-vendor move needs the same validation discipline as a cross-generation move within one vendor (numerics, quantization recalibration, backend selection), and a handful of bleeding-edge features are simply not there yet: this KB's own DSpark speculative decoding page documents that vLLM merged native CUDA support on 2026-07-01 while "ROCm support is still an open draft PR, and hardware-specific bugs were still being filed against it as of 2026-07-05."¹⁵

flowchart TB
  W["Checkpoint trained on NVIDIA<br/>(safetensors / state dict, BF16 or FP16)"]
  W -->|"loads unchanged"| N["Served on NVIDIA<br/>CUDA, NCCL, cuBLAS/CUTLASS, FlashAttention, TensorRT-LLM"]
  W -->|"loads unchanged, validate numerics"| A["Served on AMD<br/>HIP, RCCL, hipBLASLt/Composable Kernel, AITER, vLLM-ROCm/SGLang-ROCm"]
  W -->|"needs vendor-specific recalibration"| QN["NVIDIA FP8 / NVFP4 checkpoint<br/>(TensorRT-Model-Optimizer, vendor-calibrated)"]
  W -->|"needs vendor-specific recalibration"| QA["AMD Quark FP8 / MXFP4 checkpoint<br/>(amd/ org, AMD-calibrated)"]
  QN -.->|"not portable to AMD"| A
  QA -.->|"not portable to NVIDIA"| N

When to use it (and when not)¶

Use AMD/ROCm serving when capacity or economics favor it, the model's attention architecture has a confirmed, validated backend (dense GQA models are the safer starting point per the architecture-aware study above; MoE+MLA models need explicit AITER validation, not a default assumption), and a validation pass is budgeted before shipping.
Use AMD/ROCm for RL post-training when a verl (or SGLang-rollout) ROCm-built image is acceptable and its version floors (ROCm >= 7.0, vLLM > 0.11.0) and the multi-node CUDA-graph capture-size cap are tolerable, rather than requiring a fully hands-off path.
Be cautious, or wait, when the technique is bleeding-edge. Verify ROCm support per feature rather than assuming parity by default; DSpark speculative decoding (still an open ROCm draft PR while CUDA support is merged) is a concrete, already-documented example in this KB.¹⁵
Do not assume a compiled, vendor-specific artifact is portable. A TensorRT-LLM engine or an NVFP4 (Blackwell-specific microscaling) checkpoint is compiled or packed for one vendor's silicon. Only the underlying open checkpoint, the safetensors/Hugging Face format, is the portable artifact; a compiled engine built from it is not.

How to use it: engine install and backend selection¶

vLLM's ROCm wheels carry a documented trap worth calling out on its own: pre-built ROCm wheels are only available for Python 3.12. On any other Python version the installer silently falls back to the CUDA wheel from PyPI, which then fails on AMD hardware with an unrelated-looking error (libcudart.so: cannot open shared object file) rather than a clear version mismatch.¹⁴

# Check Python version first; ROCm wheels require exactly 3.12.
python3 --version

# If needed, an isolated 3.12 environment:
uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate

# Install vLLM for ROCm 7.0 (adjust the extra index for your ROCm version).
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade

The attention backend is not automatic; vLLM's ROCm selector picks an implementation depending on the model's architecture, block size, and an explicit AITER flag. An explicitly requested backend is validated against its own constraints and either used exactly as requested or rejected outright, there is no silent substitution to a different backend:⁶

Requested backend	`use_mla`	`block_size`	Outcome
`ROCM_ATTN`	`False`	unset or a multiple of 16	`ROCM_ATTN`
`ROCM_ATTN`	`False`	`1`	raises `ValueError` (block size not supported)
`ROCM_ATTN`	`True`	any	raises `ValueError` (MLA not supported)
`TRITON_MLA`	`True`	unset or a multiple of 16	`TRITON_MLA`
`TRITON_MLA`	`True`	`1`	raises `ValueError` (block size not supported)
`ROCM_AITER_MLA`	`True`	any, including `1`	`ROCM_AITER_MLA` (block-size-agnostic; explicit requests bypass `VLLM_ROCM_USE_AITER` entirely)
unspecified	`True`	unset or a multiple of 16	`ROCM_AITER_MLA` if AITER-MLA is enabled (`VLLM_ROCM_USE_AITER=1`, since `VLLM_ROCM_USE_AITER_MLA` defaults on); otherwise `TRITON_MLA`
unspecified	`True`	`1`	`ROCM_AITER_MLA` if AITER-MLA is enabled; otherwise raises `ValueError` (no valid backend)
unspecified	`False`	unset or a multiple of 16	`ROCM_ATTN`, or `ROCM_AITER_UNIFIED_ATTN`/`ROCM_AITER_FA` if `ROCM_ATTN` is excluded (for example, a KV connector is active)

A real MoE serving configuration wires this together with Quark's MXFP4 output and AMD's MoE-specific AITER kernels, for example an evaluation config serving amd/gpt-oss-20b-w-mxfp4-a-bf16:⁷

# Serve an AMD Quark MXFP4/BF16 checkpoint on ROCm with AITER-accelerated attention and MoE.
export VLLM_ROCM_USE_AITER=1
vllm serve amd/gpt-oss-20b-w-mxfp4-a-bf16 \
  --tensor-parallel-size 2 \
  --attention-backend ROCM_AITER_UNIFIED_ATTN \
  --moe-backend aiter

Why recalibrate rather than reuse another vendor's quantization constants. The following model is a controlled illustration, not a bit-exact FP8 encoder and not a reproduction of any specific vendor's real calibration data: it simulates quantizing an activation distribution with a clip range calibrated on the same data (Quark's actual workflow) versus a clip range borrowed from a different distribution standing in for another vendor's calibration set.

# quant_calibration_mismatch.py -- validated: does reusing one calibration's clip range
# for FP8 quantization on a differently-scaled activation distribution cost accuracy,
# versus recalibrating on the actual data? Run: python3 quant_calibration_mismatch.py
from __future__ import annotations

import numpy as np


def fp8_e4m3_quantize(x: np.ndarray, clip: float) -> np.ndarray:
    """Simulate symmetric FP8 E4M3-like quantization: clip to a calibrated range,
    then quantize to 2**3 = 8 mantissa steps per octave (illustrative, not a bit-exact
    E4M3 encoder). Captures the two real costs of calibration: clipping and step size."""
    x_clipped = np.clip(x, -clip, clip)
    step = clip / 448.0  # 448 is the E4M3 max normal magnitude; sets the quantization grid
    levels = 8  # illustrative mantissa resolution, not a full E4M3 bit-layout simulation
    q = np.round(x_clipped / step * levels) / levels * step
    return q


def quant_error(x: np.ndarray, clip: float) -> float:
    return float(np.mean((x - fp8_e4m3_quantize(x, clip)) ** 2))


if __name__ == "__main__":
    rng = np.random.default_rng(0)

    # Two activation distributions with the SAME shape family but different scale/outlier
    # behavior, standing in for "calibration set used to tune vendor A's checkpoint" vs
    # "the actual activations this weight sees when served on vendor B's kernel path."
    # Real transformer activations are heavy-tailed; a Student-t stand-in is illustrative.
    calib_data = rng.standard_t(df=3, size=200_000) * 1.0      # calibration-set statistics
    served_data = rng.standard_t(df=3, size=200_000) * 1.6     # actual data, 60% larger scale

    # Recalibrated: clip range derived from the SAME distribution being quantized (Quark's workflow).
    clip_matched = float(np.percentile(np.abs(served_data), 99.9))
    err_matched = quant_error(served_data, clip_matched)

    # Borrowed: clip range derived from the OTHER vendor's calibration set, then applied as-is.
    clip_borrowed = float(np.percentile(np.abs(calib_data), 99.9))
    err_borrowed = quant_error(served_data, clip_borrowed)

    assert clip_borrowed < clip_matched, "the borrowed range must under-cover the wider real data"
    assert err_borrowed > err_matched, "a mismatched calibration range must cost accuracy"

    penalty = (err_borrowed - err_matched) / err_matched
    assert penalty > 0.05, f"the mismatch penalty should be clearly non-trivial, got {penalty:.3f}"

    # Sanity: quantizing with a clip range that matches the true distribution should always
    # dominate a range calibrated on a materially different distribution, at any tested scale gap.
    for scale_gap in (1.1, 1.6, 2.5):
        served = rng.standard_t(df=3, size=50_000) * scale_gap
        matched = quant_error(served, float(np.percentile(np.abs(served), 99.9)))
        borrowed = quant_error(served, float(np.percentile(np.abs(calib_data), 99.9)))
        assert matched <= borrowed, (scale_gap, matched, borrowed)

    print(f"clip (recalibrated on served data) = {clip_matched:.3f}")
    print(f"clip (borrowed from other vendor's calib set) = {clip_borrowed:.3f}")
    print(f"quant MSE, recalibrated = {err_matched:.6f}")
    print(f"quant MSE, borrowed clip = {err_borrowed:.6f}")
    print(f"borrowed-clip error penalty = {penalty*100:.1f}% worse than recalibrating")
    print("ALL ASSERTIONS PASSED")

Executed output:

clip (recalibrated on served data) = 21.402
clip (borrowed from other vendor's calib set) = 12.794
quant MSE, recalibrated = 0.240987
quant MSE, borrowed clip = 0.530887
borrowed-clip error penalty = 120.3% worse than recalibrating
ALL ASSERTIONS PASSED

The mechanism, not the exact percentage, is the point: a clip range calibrated on the wrong distribution both under-covers the real dynamic range (clipping outliers) and wastes quantization steps on a narrower range than the data actually spans. That is what AMD Quark's own calibration pass exists to avoid, and it is why an amd/-published Quark checkpoint, not the original vendor's NVIDIA-targeted quantized checkpoint, is the supported path for FP8/MXFP4 serving on ROCm.

How to develop with it: RL post-training on AMD¶

verl's ROCm path is a distinct build, not a flag on the standard image: a ROCm/NUMA-patched Ubuntu base image with SGLang and vLLM built from source for custom gfx90a/gfx942 architecture targets, and both single-node (PPO/GRPO) and multi-node SLURM-orchestrated training supported, with vLLM as the default rollout engine and SGLang also supported.⁹ Ray's device-visibility handling is the sharp edge: launching Ray needs HIP_VISIBLE_DEVICES set alongside Ray's own RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES/RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES flags, so pin and verify against the current verl tutorial rather than assuming CUDA-style device-visibility handling carries over.⁹

Two concrete, already-documented rough edges to plan around:

vLLM sleep mode has landed on ROCm, but pin the version floor. It offloads unused GPU memory to CPU between rollout and training phases, which matters for long-context and multi-node RL; it is merged into vLLM's main branch (vLLM > 0.11.0) and needs ROCm >= 7.0, no community-patched branch required.¹⁰
Cap CUDA-graph capture sizes for multi-node ROCm deployments. Set cudagraph_capture_sizes explicitly and keep enforce_eager=False; re-check this guidance against the current verl tutorial before every vLLM bump, since it has already changed once.¹⁰

slime's fix for the ROCm/HIP pinned-memory segfault is a useful template for the general class of bug to expect: detect the platform (torch.version.hip), and override the one CUDA-safe default that is not HIP-safe, rather than patching every call site.¹¹

How to maintain it¶

Do not assume AITER is a blanket win; verify it per architecture. The controlled ablation above found a 3-5% benefit at high concurrency but 2-16x higher run-to-run variability on a dense model, "confirming that AITER's large speedups target MoE/MLA kernels specifically."⁴ Measure before and after enabling it on your actual model family.
Track vendor-published quantized checkpoints rather than hand-rolling quantization. AMD's amd/ org and Quark's calibration workflow are the supported path for FP8/MXFP4 on ROCm; re-run Quark's own calibration if you must quantize a model AMD has not already published.
Re-validate after every ROCm, vLLM, or SGLang bump, with the same discipline this KB already applies to driver/CUDA upgrades on NVIDIA: numerics (output or eval-metric comparison, not bit-exact diffing), attention-backend selection, and memory/graph-capture behavior are all points that have moved between releases in the sources cited on this page.

Running it in production¶

	What is production-composable today	What is still a rough edge
Serving	vLLM/SGLang ROCm backends; llm-d P/D disaggregation with RIXL over RoCE, validated on real hardware; Quark-quantized `amd/`-published checkpoints for known models¹²¹³	Attention-backend and offload choices are architecture-dependent, not automatic; verify per model family⁴
RL post-training	verl reproducible cross-hardware baselines (NVIDIA GPU and AMD MI300) on real RL algorithms (PPO, GRPO, ReMax, SPPO, RLOO, SPIN)⁸; vLLM sleep mode merged into main on ROCm >= 7.0¹⁰	Multi-node CUDA-graph capture needs an explicit capture-size cap and `enforce_eager=False`¹⁰
Pretraining	Full-scale MoE pretraining demonstrated on pure AMD hardware (MI300X plus Pollara networking)⁵	Ecosystem is younger than NVIDIA's; expect to write the networking/collective characterization yourself for a novel cluster shape
Bleeding-edge techniques	N/A	Verify per feature; some (DSpark speculative decoding) are CUDA-merged but ROCm-draft-PR-only as of this KB's last check¹⁵

Failure modes¶

Silent Python-version wheel fallback. Installing vLLM's ROCm wheel on anything other than Python 3.12 silently pulls the CUDA wheel instead, which then fails on AMD hardware with a confusing libcudart.so error rather than a clear version mismatch.¹⁴
Assuming AITER is a uniform win. Applying it to an attention head configuration it does not support, or expecting the same speedup magnitude on a dense model that a MoE/MLA model would see.⁴
Reusing an NVIDIA-targeted FP8/MXFP4 checkpoint's calibration constants directly instead of an AMD Quark-recalibrated amd/-published variant; the executed model above quantifies the accuracy this costs.
CUDA-graph capture crashing multi-node ROCm vLLM V1 deployments without an explicit cudagraph_capture_sizes cap and enforce_eager=False.¹⁰
Leaving non_blocking=True at its CUDA-safe default in a training or checkpointing path on ROCm/HIP, which pins memory and segfaults on a later subprocess fork.¹¹
Assuming a compiled, vendor-specific artifact is portable. A TensorRT-LLM engine or an NVFP4 checkpoint is compiled or packed for one vendor's silicon; only the underlying open checkpoint is.

Open questions & validation¶

Independently reproduce a specific MLPerf-style throughput comparison on your own MI300X/MI325X versus H100/H200 allocation; the near-parity figures reported in industry press for recent MLPerf Inference rounds were not independently re-derived from the primary MLCommons results dashboard in this research (it is JavaScript-rendered and was not fetchable as static content here), so treat them as directionally credible, not verified-exact.
Re-check verl's current ROCm tutorial for its vLLM and ROCm version floors and CUDA-graph capture guidance before every upgrade; this page's own citations already moved once (an older revision required a community-patched vLLM branch and a different CUDA-graph workaround than the current tutorial documents).
Run your own numerical-parity check (torch.allclose or an eval-metric delta, not a bit-exact diff) for the specific model you plan to move from NVIDIA to AMD serving; the LLNL study establishes that cross-vendor differences exist and are measurable in general, not what they are for your specific model and kernel path.
Whether DSpark speculative decoding, or any other bleeding-edge technique you depend on, has closed its ROCm gap since this page's last update; verify per feature rather than assuming parity.

References¶

vLLM documentation, AMD ROCm installation: https://docs.vllm.ai/en/latest/getting_started/installation/gpu.rocm.html
AMD Quark quantization toolkit (vLLM integration docs): https://docs.vllm.ai/en/v0.18.0/features/quantization/quark/
verl (RL post-training library), AMD/ROCm tutorial: https://github.com/verl-project/verl/tree/main/docs/amd_tutorial
verl, RL training baselines on GSM8k across NVIDIA GPU and AMD MI300: https://github.com/verl-project/verl/blob/main/docs/algo/baseline.md
slime (RL library), ROCm/HIP checkpoint-writer compatibility fix: https://github.com/THUDM/slime
llm-d, AMD GPU P/D disaggregation with RIXL: https://github.com/llm-d/llm-d/tree/main/guides/pd-disaggregation
Zahid, Laguna, and Le (LLNL), Testing GPU Numerics: Finding Numerical Differences Between NVIDIA and AMD GPUs (arXiv 2410.09172): https://arxiv.org/abs/2410.09172
LLM-Inference-Bench: Inference Benchmarking of Large Language Models on AI Accelerators (arXiv 2411.00136): https://arxiv.org/abs/2411.00136
AMD MI300X GPU Performance Analysis (arXiv 2510.27583): https://arxiv.org/abs/2510.27583
Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs: A Comprehensive Benchmark and Deployment Study (arXiv 2603.10031): https://arxiv.org/abs/2603.10031
Training Foundation Models on a Full-Stack AMD Platform: Compute, Networking, and System Design (arXiv 2511.17127): https://arxiv.org/abs/2511.17127
Chopper: A Multi-Level GPU Characterization Tool and Derived Insights Into LLM Training Inefficiency (arXiv 2512.08242): https://arxiv.org/abs/2512.08242
MLCommons, MLPerf Inference benchmark results (datacenter): https://mlcommons.org/benchmarks/inference-datacenter/

Zahid, Laguna, and Le, arXiv 2410.09172, abstract: "We present a study of compiler-induced numerical differences between NVIDIA and AMD GPUs... We generated more than 600,000 tests and found subtle numerical differences that come from (1) math library calls, (2) differences in floating [point behavior]." ↩
LLM-Inference-Bench, arXiv 2411.00136, abstract: "We thoroughly analyze diverse hardware platforms, including GPUs from Nvidia and AMD and specialized AI accelerators, Intel Habana and SambaNova. Our evaluation includes several LLM inference frameworks and models from LLaMA, Mistral, and Qwen families with 7B and 70B parameters." ↩
AMD MI300X GPU Performance Analysis, arXiv 2510.27583, abstract: "AMD's latest MI300X GPUs offer a compelling alternative, featuring high HBM capacity, matrix cores, and their proprietary interconnect... a comprehensive evaluation of the AMD MI300X GPUs across key performance domains critical to LLM inference including compute throughput, memory bandwidth, and interconnect communication." ↩
Architecture-Aware LLM Inference Optimization on AMD Instinct GPUs, arXiv 2603.10031, abstract: cross-architecture evaluation on 8-GPU MI325X clusters, vLLM v0.14.1, four models 235B-1T params (MoE+MLA, Dense+GQA, MoE+GQA); "MLA models require block size 1 and cannot use KV cache offloading, while GQA models benefit from both"; "the AMD AITER runtime is required for competitive MLA inference throughput and must be selectively disabled for architectures with incompatible attention head configurations"; controlled AITER ablation on Llama-3.1-405B (n=5 per condition) "reveals a modest 3-5% throughput benefit at high concurrency but 2-16x higher measurement variability, confirming that AITER's large speedups target MoE/MLA kernels specifically." ↩↩↩↩
Training Foundation Models on a Full-Stack AMD Platform, arXiv 2511.17127, abstract: "the first large-scale mixture-of-experts (MoE) pretraining study on pure AMD hardware, utilizing both MI300X GPUs and Pollara networking"; cluster/networking characterization for core collectives "to our knowledge, the first at this scale." ↩↩↩
vLLM ROCm attention-backend selection (vllm/platforms/rocm.py, RocmPlatform.get_attn_backend_cls and _get_backend_priorities, dispatched from vllm/v1/attention/selector.py; active test tests/kernels/attention/test_attention_selector.py::test_backend_selection on the main branch): an explicit request is checked only against that backend's own validate_configuration, never substituted for a different backend. TRITON_MLA requires block_size unset or a multiple of 16; ROCM_AITER_MLA accepts any block_size because its kernel always operates at page granularity 1 internally, and VLLM_ROCM_USE_AITER is not consulted for explicit requests, only for auto-selection priority when no backend is named. ↩↩
vLLM evaluation config, tests/evals/gpt_oss/configs/gpt-oss-20b-rocm-quark-mxfp4-bf16-aiter.yaml: serves amd/gpt-oss-20b-w-mxfp4-a-bf16 with ROCM_AITER_UNIFIED_ATTN attention backend, aiter MoE backend, VLLM_ROCM_USE_AITER=1, tensor-parallel-size 2. ↩
verl, docs/algo/baseline.md: reproducible RL baselines on GSM8k across models (Gemma, Qwen, DeepSeek, Mixtral), methods (SFT, PPO, GRPO, ReMax, SPPO, RLOO, SPIN), and hardware platforms (NVIDIA GPU, AMD MI300), with linked command logs and Weights & Biases runs. ↩↩
verl, docs/amd_tutorial/amd_build_dockerfile_page.rst: the ROCm build Dockerfile starts FROM a ROCm/NUMA-patched Ubuntu base image (rlsys/rocm-6.3.4-patch:rocm6.3.4-numa-patch_ubuntu-22.04) and builds SGLang and vLLM from source for custom gfx90a/gfx942 architecture targets; Ray launch needs HIP_VISIBLE_DEVICES plus the RAY_EXPERIMENTAL_NOSET_HIP_VISIBLE_DEVICES/RAY_EXPERIMENTAL_NOSET_ROCR_VISIBLE_DEVICES flags; supports single-node (PPO/GRPO) and multi-node SLURM training with both vLLM (default) and SGLang as rollout engines. ↩↩↩
verl, docs/amd_tutorial/amd_vllm_page.rst (current revision, merged 2025-11-13 via verl PR #4108, superseding an April-2025 revision that required a community-patched vLLM branch and a different CUDA-graph workaround): vLLM sleep mode is merged into vLLM's main branch for versions later than 0.11.0, recommends ROCm >= 7.0; CUDA-graph capture on multi-node AMD deployments needs actor_rollout_ref.rollout.cudagraph_capture_sizes set explicitly with enforce_eager=False. ↩↩↩↩↩↩
slime, slime/utils/rocm_checkpoint_writer.py: on ROCm/HIP, non_blocking=True during tensor operations causes tensors to be stored in pinned memory, which triggers segmentation faults when subprocesses are forked; ROCmFileSystemWriterAsync detects HIP via torch.version.hip and forces non_blocking=False. ↩↩↩
llm-d, guides/pd-disaggregation/modelserver/amd/vllm/base/patch-decode.yaml and patch-prefill.yaml (current Kustomize layout; the repo migrated its P/D guides from Helm to Kustomize in April 2026, replacing the earlier README.amd.md): serves amd/Llama-3.3-70B-Instruct-FP8-KV with --attention-backend=ROCM_ATTN required when prefill/decode tensor-parallel sizes differ; RIXL uses UCX (UCX_TLS, UCX_IB_GID_INDEX) as its KV-transfer transport. ↩↩
llm-d, guides/pd-disaggregation/modelserver/amd/vllm/oci/kustomization.yaml: single-node deployment on OCI OKE BM.GPU.MI300X.8, one TP=4 decode pod plus four TP=1 prefill pods, ghcr.io/llm-d/llm-d-rocm:v0.7.0 container image (guides/recipes/modelserver/components/images/amd-vllm/kustomization.yaml). ↩↩
vLLM ROCm installation docs (docs/getting_started/installation/gpu.rocm.inc.md): "vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1"; "ROCm pre-built wheels are only available for Python 3.12. If you are using a different Python version... the installer will silently fall back to the CUDA wheel from PyPI, which will fail on AMD GPUs with errors like libcudart.so: cannot open shared object file." ↩↩
This KB, DSpark speculative decoding: vLLM merged native dspark speculative-decoding support 2026-07-01; "ROCm support is still an open draft PR, and hardware-specific bugs were still being filed against it as of 2026-07-05." ↩↩↩