Markdown

vLLM Transformers modeling backend¶

Scope: serving a Hugging Face transformers model in vLLM with --model-impl transformers, without writing a vLLM model. Published Qwen3 results reached native-backend throughput on one benchmark; parity on other architectures and latency regimes remains unverified. Covers the torch.fx plus AST fusion pass added in vLLM PR #47187, the three fusers (GLU, QKV, RMSNorm) and the merged-projection sharding they depend on, how to prove the fusions actually fired, the published benchmark and the regimes it does not cover, the rules a model must follow to be fusable, and the hard limits (linear attention, non-compliant remote code). For model-by-model deployment recipes see serving open-weight models.

What it is¶

The Transformers modeling backend is a vLLM code path that takes the model definition from the transformers library and runs it inside the vLLM engine. It is selected with one flag, --model-impl transformers. vLLM keeps everything that makes it fast at the engine level: continuous batching, the paged KV cache, custom attention kernels, torch.compile, and CUDA graphs. Only the modeling code changes hands.

The backend itself is not new; it landed in April 2025. What it used to do was treat attention as the only bottleneck worth fixing: it swapped vLLM's attention implementation in at runtime and left the rest of the module tree exactly as transformers wrote it. Everything else a hand-written vLLM port does (fusing sibling projections into one matmul, mapping MoE experts onto grouped kernels, deriving tensor-parallel and pipeline-parallel plans) was absent. Authors who wanted the last few points of throughput still wrote and maintained a second implementation.

vLLM PR #47187, "Make the Transformers modeling backend as fast as native vLLM" (merged 2026-07-06, 2,679 lines added across 23 files), closes that gap. The implementation itself is 12 Python files (1,620 of those lines) under vllm/model_executor/models/transformers/; the remainder of the diff is CI config, CODEOWNERS, docs, and tests under tests/models/transformers/, with no compiled-code (CUDA/C++/build) changes anywhere in the PR. At load time the backend traces each module class with torch.fx, matches the resulting graph against a small set of known patterns, and then uses ast to rewrite that module's forward source in place so it calls vLLM's fused kernels. The rewritten module keeps its class and its unconsumed attributes, and is still handed to torch.compile and CUDA graphs like any native vLLM model.

Why use it¶

Earlier serving coverage. A compliant model merged into transformers can run before a native vLLM port exists. Native-equivalent throughput is established only for the published Qwen3 cases.
One definition for training and inference. vLLM model implementations are inference-only. transformers implementations train. Using the same model definition for training, evaluation, and RL rollouts reduces model-definition skew that is otherwise invisible until a reward curve misbehaves.
No second implementation to keep in sync. Two independent ports of one architecture drift, and the drift is a correctness risk that surfaces as a quality regression rather than a crash.

When to use it (and when not)¶

Situation	Backend
Model is in `transformers`, has no native vLLM implementation yet	`transformers`
You are the model author and want one codebase for train, eval, RL rollout, and serve	`transformers`
Model uses linear attention (Mamba-style and hybrid state-space blocks)	Not supported yet, use a native port
Custom architecture whose code lives in a Hub repo	Only if written compliantly (see below); otherwise not supported
Mature native vLLM implementation, and your workload differs from the published benchmark	Measure both, do not assume parity transfers

The parity claim is specific. It was established on three Qwen3 models, on one workload, on one node. Qwen3 dense and MoE blocks are close to the architectures vLLM's native implementations were tuned for, and they are exactly the shapes the fusers match best. Treat parity on a materially different architecture, or in a materially different serving regime, as unverified until measured.

Architecture¶

flowchart TD
  A["HF checkpoint"] --> B{"--model-impl"}
  B -->|vllm| C["hand-written vLLM model"]
  B -->|transformers| D["transformers modeling code"]
  D --> E["torch.fx trace, once per module class"]
  E --> F{"pattern match"}
  F -->|"act(gate(x)) * up(x)"| G["gate_up_proj (MergedColumnParallelLinear)"]
  F -->|"q(x), k(x), v(x)"| H["qkv_proj (QKVParallelLinear)"]
  F -->|"RMSNorm leaf module"| I["vLLM RMSNorm"]
  F -->|"no match"| J["left unfused, still runs, slower"]
  G --> K["ast rewrite of forward, recompile"]
  H --> K
  I --> K
  K --> L["torch.compile and CUDA graphs"]
  C --> L
  J --> L
  L --> M["vLLM engine: continuous batching, paged KV cache"]

How to use it¶

One flag, composed with the usual parallelism options. These are unexecuted reference templates read at vLLM merge commit 5bce653e09ca62c870ea18d01a4180dc48d3bacb; pin a release or commit plus model revisions and re-measure on your own hardware.

# Qwen3-4B dense, single GPU
vllm serve Qwen/Qwen3-4B --model-impl transformers

# Qwen3-32B dense, tensor-parallel across 2 GPUs
vllm serve Qwen/Qwen3-32B --model-impl transformers --tensor-parallel-size 2

# Qwen3-235B-A22B-FP8 MoE, data-parallel plus expert-parallel across 8 GPUs
vllm serve Qwen/Qwen3-235B-A22B-FP8 --model-impl transformers \
  --data-parallel-size 8 --enable-expert-parallel --max-model-len 8192

--model-impl takes auto (native if one exists, else transformers), vllm, or transformers. Passing transformers explicitly is what forces the comparison, and what a benchmark must do.

Confirm which implementation actually loaded, rather than assuming:

from vllm import LLM
llm = LLM(model="Qwen/Qwen3-4B", model_impl="transformers")
llm.apply_model(lambda model: print(type(model)))  # Transformers... means the backend is active

Then confirm the expected fusion signatures fired. Each successful fusion emits a log line; capture startup logs and count by fusion type:

# Start the server under the deployment's normal log collector. For a local check:
VLLM_LOGGING_LEVEL=INFO vllm serve Qwen/Qwen3-4B --model-impl transformers 2>&1 | tee server.log

# Run after startup in another shell. Prefixes such as timestamps make '^Fused:' unsafe.
rg -c 'Fused:.*gate_up_proj' server.log
rg -c 'Fused:.*qkv_proj' server.log

A zero count for an expected signature means that rewrite did not fire. Record the expected count per module class and model depth in CI; a generic non-zero total can miss a partially unfused model.

How to develop with it: the fusion contract¶

The backend matches three patterns, in this order, per module class.

Fuser	Pattern it matches	What it becomes	Shard ids
`GLUFuser`	`act(gate(x)) * up(x)`	`gate_up_proj`, a `MergedColumnParallelLinear`, with the activation swapped for its fused `AndMul` form (for example `SiluAndMul`)	`gate`=0, `up`=1
`QKVFuser`	`q(x), k(x), v(x)`	`qkv_proj`, a `QKVParallelLinear`	`q`, `k`, `v`
`RMSNormFuser`	a leaf module doing raw RMSNorm tensor math	vLLM's `RMSNorm`	none

Tracing is skipped entirely for any module with fewer than two nn.Linear children that is not a leaf, so the pass is cheap on large module trees. If a fuser matches but the forward cannot be rewritten, the exception is caught, the module is left unfused, and the model still runs. Fusion failure is a performance event, not a crash, which is precisely why it needs to be asserted rather than assumed.

For a custom model to be fusable it must follow the upstream compliance rules: pass kwargs down from MyModel to MyAttention, call attention through ALL_ATTENTION_FUNCTIONS, and set _supports_attention_backend = True. MoE models additionally need the sparse block to expose an experts attribute whose class either inherits nn.ModuleList or holds 3D nn.Parameters, with a forward accepting hidden_states, top_k_index, and top_k_weights. The blog's warning that Hub remote-code models are "unlikely to work" is a statement about compliance, not a prohibition: a remote-code model written to these rules does work, and most existing ones were not written to them.

Why the merged projections carry shard ids¶

Fusing is supposed to be a pure performance transform. Sharding the fused weight is where it can silently corrupt, because a merged projection is a concatenation of independent projections, not one large matrix. That is the reason each fuser carries a shard id per source projection instead of merging and forgetting. The following is an executed, scaled analogue of Qwen3-32B. It preserves the unequal 8:1:1 Q:K:V row ratio while avoiding the multi-gigabyte allocation that the real dimensions would create in NumPy.

# fused_projection_sharding.py -- validated. What vLLM's Transformers-backend fusers must get
# right when they rewrite q/k/v into one QKVParallelLinear and gate/up into one
# MergedColumnParallelLinear. Fusion must remain numerically equivalent. Sharding the fused
# weight is where a right-shaped result can silently become wrong.
import numpy as np

rng = np.random.default_rng(0)

# Scaled analogue of Qwen3-32B. Real config: H=5120, NH=64, NKV=8, D=128, I=25600.
# The smaller tensors retain the real q:k:v row ratio of 8:1:1 and TP=2 divisibility.
H, NH, NKV, D, I = 80, 16, 2, 4, 160
TP = 2

x = rng.standard_normal((4, H), dtype=np.float32)          # 4 tokens
Wq = rng.standard_normal((NH * D, H), dtype=np.float32) * 0.2
Wk = rng.standard_normal((NKV * D, H), dtype=np.float32) * 0.2
Wv = rng.standard_normal((NKV * D, H), dtype=np.float32) * 0.2
Wg = rng.standard_normal((I, H), dtype=np.float32) * 0.2   # gate_proj
Wu = rng.standard_normal((I, H), dtype=np.float32) * 0.2   # up_proj

silu = lambda z: z / (1.0 + np.exp(-z))

print(f"scaled Qwen3-32B analogue: hidden={H} q_heads={NH} kv_heads={NKV} head_dim={D} intermediate={I}")
print(f"merged qkv rows = {NH*D} + {NKV*D} + {NKV*D} = {NH*D + 2*NKV*D}  "
      f"(q:k:v = {NH*D}:{NKV*D}:{NKV*D}, NOT three equal parts)")
print(f"merged gate_up rows = {I} + {I} = {2*I}\n")

# --- 1. Fusion on one GPU is numerically equivalent ------------------------------------------
q_ref, k_ref, v_ref = x @ Wq.T, x @ Wk.T, x @ Wv.T
Wqkv = np.concatenate([Wq, Wk, Wv], axis=0)                 # the merged weight
qkv = x @ Wqkv.T
q_f, k_f, v_f = np.split(qkv, [NH * D, NH * D + NKV * D], axis=1)
d_qkv = max(np.abs(q_f - q_ref).max(), np.abs(k_f - k_ref).max(), np.abs(v_f - v_ref).max())

glu_ref = silu(x @ Wg.T) * (x @ Wu.T)
Wgu = np.concatenate([Wg, Wu], axis=0)
g_f, u_f = np.split(x @ Wgu.T, [I], axis=1)
d_glu = np.abs(silu(g_f) * u_f - glu_ref).max()
print("[1] fusion, single GPU (TP=1)")
print(f"    qkv     max|fused - unfused| = {d_qkv:.3e}")
print(f"    gate_up max|fused - unfused| = {d_glu:.3e}")
assert d_qkv < 1e-4 and d_glu < 1e-4, "fusion must not change the result"
print("    OK: fusing is a pure performance transform, the math is unchanged\n")

# --- 2. Sharding the merged weight: correct vs naive -----------------------------------------
# Correct (what QKVParallelLinear/MergedColumnParallelLinear do): slice EACH source projection
# by rank, then concatenate. Rank r owns q-heads, k-heads and v-heads r.
def shard_correct(rank):
    qs, ks, vs = NH * D // TP, NKV * D // TP, NKV * D // TP
    return np.concatenate([Wq[rank*qs:(rank+1)*qs],
                           Wk[rank*ks:(rank+1)*ks],
                           Wv[rank*vs:(rank+1)*vs]], axis=0)

# Naive (a plain ColumnParallelLinear over the merged matrix): one contiguous row block.
def shard_naive(rank):
    rows = Wqkv.shape[0] // TP
    return Wqkv[rank*rows:(rank+1)*rows]

def run(shard_fn):
    qs, ks = NH * D // TP, NKV * D // TP
    qo, ko, vo = [], [], []
    for r in range(TP):
        y = x @ shard_fn(r).T
        a, b, c = np.split(y, [qs, qs + ks], axis=1)
        qo.append(a); ko.append(b); vo.append(c)
    return np.concatenate(qo, 1), np.concatenate(ko, 1), np.concatenate(vo, 1)

print(f"[2] QKV under tensor parallelism (TP={TP})")
for name, fn in (("correct (per-shard slice)", shard_correct), ("naive (contiguous split)", shard_naive)):
    q_t, k_t, v_t = run(fn)
    err = max(np.abs(q_t - q_ref).max(), np.abs(k_t - k_ref).max(), np.abs(v_t - v_ref).max())
    bad = np.mean(~np.isclose(np.concatenate([q_t, k_t, v_t], 1),
                              np.concatenate([q_ref, k_ref, v_ref], 1), atol=1e-4)) * 100
    print(f"    {name:26s} max err = {err:.3e}   elements wrong = {bad:5.1f}%")
    if fn is shard_correct:
        assert err < 1e-4, "per-shard slicing must reproduce the unsharded result"
    else:
        assert err > 1.0 and bad > 50, "contiguous split must be visibly wrong"

rows = Wqkv.shape[0] // TP
print(f"    rank0's contiguous block = rows 0:{rows} = q-heads 0:{rows//D} only. It holds no k "
      f"and no v rows,\n    so what it calls 'k' and 'v' are really more q. No exception is "
      f"raised; the output is just wrong.\n")

# --- 3. Same trap on gate_up, and the error survives the elementwise product ------------------
print(f"[3] gate_up under tensor parallelism (TP={TP})")
gu_correct = [np.concatenate([Wg[r*(I//TP):(r+1)*(I//TP)], Wu[r*(I//TP):(r+1)*(I//TP)]], 0) for r in range(TP)]
gu_naive = [Wgu[r*(2*I//TP):(r+1)*(2*I//TP)] for r in range(TP)]
for name, shards in (("correct (per-shard slice)", gu_correct), ("naive (contiguous split)", gu_naive)):
    outs = []
    for W in shards:
        g, u = np.split(x @ W.T, [W.shape[0] // 2], axis=1)
        outs.append(silu(g) * u)
    y = np.concatenate(outs, axis=1)
    err = np.abs(y - glu_ref).max()
    print(f"    {name:26s} max err = {err:.3e}")
    if shards is gu_correct:
        assert err < 1e-4
    else:
        assert err > 1.0
print("    rank0's contiguous block is the whole gate matrix, so it computes silu(gate_a)*gate_b:")
print("    an activation multiplied by another slice of ITSELF, never touching up_proj.\n")

# --- 4. The edge case that makes the shard ids load-bearing: TP > num_kv_heads ----------------
NKV_235 = 4   # Qwen3-235B-A22B has 4 kv heads
print("[4] Qwen3-235B-A22B (4 kv heads): what happens as TP grows")
for tp in (2, 4, 8):
    if NKV_235 % tp == 0:
        print(f"    TP={tp}: {NKV_235 // tp} kv head(s)/rank, split cleanly")
    else:
        print(f"    TP={tp}: {NKV_235}/{tp} = {NKV_235/tp} kv heads/rank is not an integer -> "
              f"vLLM must REPLICATE kv heads across ranks")
assert NKV_235 % 8 != 0
print("    A shard id per projection is what lets vLLM replicate k and v while still splitting q.")
print("    A single contiguous split of the merged matrix has no way to express that.\n")

print("All assertions passed.")

Executed output:

scaled Qwen3-32B analogue: hidden=80 q_heads=16 kv_heads=2 head_dim=4 intermediate=160
merged qkv rows = 64 + 8 + 8 = 80  (q:k:v = 64:8:8, NOT three equal parts)
merged gate_up rows = 160 + 160 = 320

[1] fusion, single GPU (TP=1)
    qkv     max|fused - unfused| = 0.000e+00
    gate_up max|fused - unfused| = 0.000e+00
    OK: fusing is a pure performance transform, the math is unchanged

[2] QKV under tensor parallelism (TP=2)
    correct (per-shard slice)  max err = 0.000e+00   elements wrong =   0.0%
    naive (contiguous split)   max err = 6.704e+00   elements wrong =  55.0%
    rank0's contiguous block = rows 0:40 = q-heads 0:10 only. It holds no k and no v rows,
    so what it calls 'k' and 'v' are really more q. No exception is raised; the output is just wrong.

[3] gate_up under tensor parallelism (TP=2)
    correct (per-shard slice)  max err = 0.000e+00
    naive (contiguous split)   max err = 2.206e+01
    rank0's contiguous block is the whole gate matrix, so it computes silu(gate_a)*gate_b:
    an activation multiplied by another slice of ITSELF, never touching up_proj.

[4] Qwen3-235B-A22B (4 kv heads): what happens as TP grows
    TP=2: 2 kv head(s)/rank, split cleanly
    TP=4: 1 kv head(s)/rank, split cleanly
    TP=8: 4/8 = 0.5 kv heads/rank is not an integer -> vLLM must REPLICATE kv heads across ranks
    A shard id per projection is what lets vLLM replicate k and v while still splitting q.
    A single contiguous split of the merged matrix has no way to express that.

All assertions passed.

Three results carry over to production. Fusion should preserve logits within dtype- and kernel-appropriate tolerances; exact equality is not a portable GPU requirement. Grouped-query attention makes the merged QKV rows unequal (8192:1024:1024 on Qwen3-32B), so any code that "splits the merged matrix into three" is wrong before tensor parallelism is even considered. And a contiguous split of a merged weight across ranks raises no exception while corrupting most of the output, which is why orig_to_new_stacked maps every checkpoint tensor to an explicit (merged_name, shard_id) pair, keyed by qualname so that an unfused MoE expert's own gate_proj is never caught by the same rule.

How to maintain it¶

The failure mode of this feature is silence. A fuser that stops matching returns None, the module is left unfused, and the model serves correct tokens more slowly. A transformers refactor that renames a projection, wraps an activation, or moves the multiply can therefore cost throughput on a version bump with no error and no test failure.

Pin vllm and transformers together, and treat a bump of either as a performance-affecting change.
Assert expected fusion signatures and counts in CI from captured startup logs. Match Fused: anywhere on the line because the logger may prepend timestamps or ranks.
Watch for the one loud signal the backend does emit. A class whose name ends in RMSNorm that does not match structurally logs looks like an RMSNorm but its computation did not match the expected pattern, so it was left unfused. That warning is a direct report of lost performance.
Quantized models depend on packed_modules_mapping to unpack a fused layer back into its per-shard quantization configs. A new fusion plus an unusual quantization scheme is the combination most likely to need upstream work.

How to run it in production¶

The published benchmark is narrow, and reading it precisely is the difference between a safe migration and a surprise.

Model	Parallelism	Before PR	After PR	vs pre-PR	% of native
Qwen3-4B (bf16)	TP1	46,850 tok/s	47,443 tok/s	+1.3%	100.0%
Qwen3-32B (bf16)	TP2	14,310 tok/s	14,560 tok/s	+1.7%	100.1%
Qwen3-235B-A22B-FP8 (MoE)	DP8 + EP	31,382 tok/s	33,152 tok/s	+5.6%	102.0%

Measured on 8x H100 80GB, with 1024-token prompts and 128-token generations, 1000 prompts from the random dataset. The dense models use the offline vllm bench throughput harness; the MoE model uses online vllm bench serve. The before/after delta is obtained by checking out only vllm/model_executor/models/transformers/ at the PR's parent commit, so nothing else moves.

What this does and does not establish:

The headline is parity, and parity holds on these three models. The improvement the PR delivers on them is small, because the backend was already at 96.6% to 98.7% of native before it. The fusers close a gap of one to three points and, on the MoE model, overshoot.
The benchmark script contains one invocation per configuration and reports no repeat loop or variance. The 102% MoE result may reflect noise or a genuine implementation difference; one point cannot bound harness precision. Treat the small dense-model deltas as unresolved without repeated runs. The parity conclusion survives this; the per-model improvement figures do not.
Throughput at saturation is the only metric. With 1024-token prompts against 128-token generations, this workload is prefill-heavy and batch-saturated, which is the regime that most amortizes per-layer Python overhead. Time to first token, inter-token latency, and low-concurrency behaviour are not measured. If you serve an interactive, latency-bound endpoint, benchmark TTFT and ITL yourself before switching.
MoE with expert parallelism shows the largest reported delta (+5.6%). A plausible explanation is the many-to-one mapping onto grouped expert kernels, but the benchmark does not isolate that cause. See expert parallelism.

The safe migration is an A/B on your own traffic: serve the same checkpoint under --model-impl vllm and --model-impl transformers, assert the fusion count is non-zero on the second, and compare both throughput and tail latency before making it the default.

Failure modes¶

Symptom	Cause	Action
Throughput well below native, no errors	Fusers did not match; model runs unfused	count expected `Fused:` signatures in the captured startup log; a missing signature identifies the unfused path
`looks like an RMSNorm but its computation did not match`	Norm class not structurally recognised	Expect lost performance; align the norm with the upstream pattern or accept the gap
Model fails to load with remote code	Custom model not written to the backend's compliance rules	Add `_supports_attention_backend`, route attention through `ALL_ATTENTION_FUNCTIONS`, thread `kwargs`
Linear-attention or hybrid state-space model rejected	Not supported as of PR #47187 (2026-07-06), stated upstream as planned	Use a native implementation
Wrong outputs only at TP>1	Merged weight sharded contiguously rather than per shard id	The validated failure above; a real bug in custom sharding code, never silent-safe
Quantized fused layer fails to load	`packed_modules_mapping` cannot unpack the scheme	Check the fused layer's per-shard quantization configs upstream

Open questions and validation¶

Parity is established for Qwen3-shaped dense and MoE decoders. It is not established for architectures whose blocks differ structurally from what the three fusers match, and the pre-PR gap on such a model could be much larger than the one to three points seen here. Measure, do not extrapolate.
No latency-regime numbers are published. The claim that the backend is at parity for interactive serving is currently unverified in either direction.
Linear attention support is stated as coming. Re-check the vLLM support matrix rather than trusting this page's snapshot.

References¶

Native-speed vLLM transformers modeling backend, Harry Mellor and Lysandre Debut, Hugging Face, 2026-07-08.
vLLM PR #47187: Make the Transformers modeling backend as fast as native vLLM, merged 2026-07-06.
Benchmark runner used for the published figures.
vLLM docs: Transformers modeling backend and writing custom models.
Transformers backend integration in vLLM, the original April 2025 integration.
torch.fx and the ast module, the two mechanisms the fusion pass is built on.