Markdown

Block-wise FP8 quantization for VLA/VLM¶

Scope: how to post-training-quantize a small Vision-Language-Action or vision-language model to FP8 without losing task accuracy, using fine-grained block-wise scaling (independent scale per 128x128 block) and a mixed-precision split that quantizes the language backbone but keeps the vision tower (ViT) in high precision. Distilled from the JD Technology embodied-infrastructure report (Guo et al., 2026), which applies this to Qwen2.5-VL-3B for >140% inference compute speedup, 36.6% compression, and held GSM8K/MMLU accuracy. Sits alongside quantization for inference, NVFP4, and tensor cores and mixed precision.

The numpy block labelled core math is self-contained, runnable, and validated. It builds the real OCP E4M3 grid and demonstrates the load-bearing, sometimes-surprising result: for FP8 (a floating-point format) block-wise scaling is nearly a no-op on round-trip error, which is exactly why the report reports maintained accuracy; the large win of fine-grained scaling is for fixed-point INT8. The torch/LLM-Compressor snippets are reference templates: pin versions, validate before production. One industrial source; re-measure on your model.

What it is¶

Quantization stores weights (and sometimes activations) in fewer bits to shrink the model and speed up matmuls. FP8 uses an 8-bit floating-point format, almost always E4M3 (1 sign bit, 4 exponent bits, 3 mantissa bits, exponent bias 7), whose largest finite value is 448 and which has no infinities. Two design choices define this recipe:

Block-wise (fine-grained) scaling. A quantized tensor is scaled before rounding so its values land in the format's representable range. The granularity of that scale is the knob: one scale for the whole tensor (per-tensor), one per row/channel (per-channel), or one per fixed-size block. This recipe partitions the last two dimensions into 128x128 blocks and gives each block its own scale (the same grouping DeepSeek-V3 popularized for FP8 GEMM). Finer granularity isolates outliers to their own block instead of letting one large value set a coarse scale for everything.
Mixed-precision module split. The vision module (ViT) is not quantized, preserving visual-feature quality, while the language module (LLM) gets block-wise FP8. Quantization is applied post-training (PTQ), with no FP8-aware fine-tuning (no QAT).

The target is small models. VLA and the VLMs they embed are typically 0.5B to 3B parameters because they run under real-time and edge constraints, and small models are exactly where a careless per-tensor FP8 scale can cost accuracy.

Why use it¶

Compression and speed with accuracy parity. The report measured 36.6% model compression and over 140% inference compute speedup while GSM8K and MMLU accuracy held. The compression figure implies about 73% of parameters were quantized 16-bit to 8-bit (validated below), consistent with a 3B VLM whose ViT is roughly a quarter of the weights and stays in high precision.
Block-wise beats per-tensor for the general case, and never loses. Isolating outliers to a 128x128 block keeps a single large weight from forcing a coarse scale across an entire tensor. For fixed-point this is a several-fold error reduction; for FP8 it is a safety margin (see the honest caveat below).
PTQ is cheap. No retraining, no QAT loop; a calibration pass suffices, so the recipe is a deployment step, not a training project.
It beat the alternatives the report tested. Against AWQ, GPTQ-int4, and LLM-Compressor FP8-dynamic on GSM8K/MMLU, the report found block-wise FP8 best overall on the joint speed-and-accuracy axis for these small models.

The honest caveat: why block-wise helps FP8 less than INT8¶

The intuitive story ("fine-grained scaling isolates outliers, so it must lower quantization error") is true for INT8 and largely false for FP8, and the difference is worth understanding before you attribute a win to the wrong cause.

FP8 is a floating-point format: its exponent adapts to each value's magnitude, so relative precision is roughly constant (about 3 mantissa bits) regardless of scale. A symmetric amax round-trip is therefore nearly scale-invariant: per-tensor and block-wise FP8 produce almost identical reconstruction error (the core-math block measures them within 0.3% of each other). That is precisely why the report reports accuracy maintained, not improved.

INT8 is fixed-point: a constant absolute step across all magnitudes. One outlier forces a coarse step, and every small value is crushed to a few levels. Block-wise scaling localizes that damage to the outlier's own block, cutting error several-fold (the core-math block measures about 4x). Per-tensor INT8 is more than an order of magnitude worse than either FP8 path, which is the real reason FP8 is the robust default for small models: its floating exponent already survives outliers without needing fine-grained scaling.

So block-wise FP8's value is mostly hardware-shaped (per-block scales enable efficient FP8 GEMM with bounded accumulation error) plus a correctness safety margin, not a tensor-RMSE improvement. Claiming block-wise "fixes FP8 accuracy" overstates it; the accurate claim is that FP8 is already robust and block-wise keeps it that way while enabling fast grouped matmuls.

When to use it (and when not)¶

Use block-wise FP8 PTQ for small (0.5B to 3B) VLA/VLM language backbones you want to deploy under latency or edge constraints, when you can tolerate a calibration pass but not a retraining loop.
Keep the vision tower in higher precision unless you have measured that quantizing it holds accuracy; the report deliberately does not quantize the ViT.
Prefer INT4 methods (AWQ, GPTQ) when memory is the hard constraint and you can absorb their accuracy/latency profile; they compress more than 8-bit but the report found FP8 better on the joint metric for its small models.
Do not expect a tensor-accuracy improvement from block granularity on FP8. If your goal is only lower weight-reconstruction error on FP8, per-tensor is essentially as good; choose block-wise for GEMM efficiency and outlier safety, or when you may switch to a fixed-point format.
Reach for QAT instead of PTQ only if PTQ measurably regresses your task; the report achieved parity without it.

Architecture¶

The scale granularity is the whole story. Per-tensor uses one scalar for the entire matrix; per-channel one per output channel; block-wise one per 128x128 tile. The vision and language modules are quantized independently (or not at all, for the ViT).

flowchart TB
  subgraph MODEL["VLA / VLM"]
    VIT["Vision tower (ViT): kept high precision"]
    LLM["Language backbone: block-wise FP8 PTQ"]
  end
  subgraph QUANT["Block-wise FP8 of one weight matrix"]
    W["weight matrix"] --> TILE["partition into 128x128 blocks"]
    TILE --> SC["per-block amax scale -> E4M3 (max 448)"]
    SC --> GEMM["FP8 grouped GEMM, per-block scales"]
  end
  LLM --> W

Core math (runnable): E4M3 grid, FP8 vs INT8, and the compression model¶

This numpy block builds the OCP E4M3 representable grid, quantizes a weight tensor that has a couple of large local outliers, and measures the truth: FP8 per-tensor and block-wise are a wash, INT8 block-wise is several times better than per-tensor, and per-tensor INT8 is an order of magnitude worse than FP8. It also checks the zero-block edge, exact round-trip on representable values, and the 36.6%-compression relationship. Run: python3 fp8_blockwise.py.

# fp8_blockwise.py -- fine-grained (block-wise) FP8 quantization, core math. numpy only.
import numpy as np

# ---------- OCP E4M3 grid (bias 7, no inf; (E=15,M=7) is NaN, excluded) ----------
def e4m3_grid():
    vals = {0.0}
    for E in range(1, 16):                                     # normals
        for M in range(8):
            if E == 15 and M == 7:                             # reserved NaN, not a number
                continue
            vals.add((1 + M / 8) * 2.0 ** (E - 7))
    for M in range(1, 8):                                      # subnormals (E=0)
        vals.add((M / 8) * 2.0 ** (1 - 7))
    return np.array(sorted(vals))

GRID = e4m3_grid()
FP8_MAX = GRID[-1]
assert FP8_MAX == 448.0                                        # E4M3 max normal
assert abs(GRID[GRID > 0][0] - 2.0 ** -9) < 1e-12              # min positive subnormal = 2^-9

def to_fp8(x):                                                 # round magnitude to nearest grid point
    s = np.sign(x); m = np.abs(x)
    idx = np.clip(np.searchsorted(GRID, m), 0, len(GRID) - 1)
    lo = np.clip(idx - 1, 0, len(GRID) - 1)
    pick_lo = np.abs(m - GRID[lo]) <= np.abs(GRID[idx] - m)
    return s * np.where(pick_lo, GRID[lo], GRID[idx])

def scale_amax(block, qmax):                                  # per-block amax scale (guard zero block)
    a = np.abs(block).max()
    return (a / qmax) if a > 0 else 1.0

def qd_fp8(x, scale):
    return to_fp8(np.clip(x / scale, -FP8_MAX, FP8_MAX)) * scale

def qd_int8(x, scale):                                        # symmetric INT8: constant absolute step
    return np.clip(np.round(x / scale), -127, 127) * scale

def blockwise(W, qd, qmax, bs):
    out = np.empty_like(W)
    for i in range(0, W.shape[0], bs):
        for j in range(0, W.shape[1], bs):
            blk = W[i:i+bs, j:j+bs]
            out[i:i+bs, j:j+bs] = qd(blk, scale_amax(blk, qmax))
    return out

def per_tensor(W, qd, qmax):
    return qd(W, scale_amax(W, qmax))

def rmse(a, b):
    return float(np.sqrt(np.mean((a - b) ** 2)))

# Round-trip exact on representable values; zero block is safe (no div-by-zero, exact 0).
exact = np.array([0.0, 1.0, 2.0, 448.0, 2.0 ** -9, 1.75 * 2 ** 8])
assert np.allclose(to_fp8(exact), exact)
qz = blockwise(np.zeros((128, 128)), qd_fp8, FP8_MAX, 128)
assert not np.isnan(qz).any() and np.all(qz == 0.0)

# A weight tensor with a couple of large, LOCAL outliers (16 blocks of 128x128).
rng = np.random.default_rng(0)
W = rng.normal(0, 0.02, size=(512, 512))
W[10, 20] = 6.0; W[11, 21] = -5.0                            # ~300x the typical magnitude, one block

# FP8: floating exponent adapts -> per-tensor and block-wise agree within noise (scale-invariant).
e_fp8_t = rmse(W, per_tensor(W, qd_fp8, FP8_MAX))
e_fp8_b = rmse(W, blockwise(W, qd_fp8, FP8_MAX, 128))
assert abs(e_fp8_b - e_fp8_t) / e_fp8_t < 0.05               # a wash: WHY FP8 accuracy is "maintained"

# INT8: fixed step. Outlier forces a coarse per-tensor step; block-wise localizes it (~4x lower RMSE).
e_int8_t = rmse(W, per_tensor(W, qd_int8, 127))
e_int8_b = rmse(W, blockwise(W, qd_int8, 127, 128))
assert e_int8_b < 0.3 * e_int8_t                             # block-wise cuts INT8 error several-fold
assert e_int8_t > 10 * e_fp8_t                               # per-tensor INT8 >> FP8: FP8 is the robust default

# Compression model: 36.6% size cut => fraction of params quantized 16->8 bit (0.5x each).
f_needed = 0.366 / 0.5
assert abs(f_needed - 0.732) < 1e-9                          # ~73% of params must be the quantized LLM
vit, llm = 0.8, 2.2                                          # a 3B VLM: ~0.8B ViT (kept) + ~2.2B LLM
assert abs((llm / (vit + llm)) * 0.5 - 0.366) < 0.01        # matches the reported 36.6%

print("PASS:",
      f"FP8 t={e_fp8_t:.5f} b={e_fp8_b:.5f} (wash) | INT8 t={e_int8_t:.4f} b={e_int8_b:.5f}"
      f" ({e_int8_t/e_int8_b:.1f}x) | 36.6% => {f_needed*100:.1f}% params quantized")

How it works¶

For each 128x128 block, the amax scale maps the block's largest magnitude to the top of the E4M3 range (448), every value is divided by that scale, rounded to the nearest representable E4M3 grid point, and the scale is kept alongside the FP8 tensor for dequantization at matmul time. Because each block owns its scale, a big weight in one block cannot force a coarse grid on another block. The ViT is skipped entirely, so visual features never pass through 8-bit rounding.

At inference the FP8 tensors feed a grouped GEMM that applies per-block scales during accumulation, which is where the throughput comes from: FP8 tensor-core matmuls run at roughly double the rate of BF16 on supported hardware (tensor cores and mixed precision), and the per-block scaling keeps accumulation error bounded so the speed does not cost accuracy. The core-math block is the load-bearing evidence for the accuracy claim: it shows FP8's round-trip error is scale-invariant, so the block granularity is about GEMM efficiency and outlier safety rather than a lower reconstruction error.

How to use it¶

Reference template using LLM Compressor for block-wise FP8 PTQ (pin versions; validate on your model):

# REFERENCE TEMPLATE (needs llmcompressor + transformers; not executed here).
# Core math is validated in numpy above.
from llmcompressor.transformers import oneshot
from llmcompressor.modifiers.quantization import QuantizationModifier

recipe = QuantizationModifier(
    targets="Linear",
    scheme="FP8_BLOCK",              # 128x128 block-wise FP8; verify the exact scheme name for your version
    ignore=["re:.*visual.*", "lm_head"],   # keep the ViT and the output head out of FP8
)
oneshot(model="Qwen/Qwen2.5-VL-3B-Instruct", recipe=recipe,
        dataset="open_platypus", output_dir="./qwen2.5-vl-3b-fp8-block")  # calibration pass

Exclude the vision modules and the LM head from quantization via name patterns, matching the report's ViT-in-high-precision split.
Calibrate on representative data. PTQ needs a calibration set that reflects deployment inputs so per-block scales are set from realistic magnitudes.
Confirm the target hardware supports FP8 tensor cores (Hopper/Blackwell class); on hardware without native FP8 matmul the compression is real but the speedup is not.

How to develop with it¶

Start from per-tensor FP8 as the baseline, then add block granularity. Because FP8 is scale-invariant on round-trip, measure whether block-wise actually changes your downstream metric; if it does not, you are adopting it for GEMM efficiency and safety, which is a legitimate but different reason.
Sweep the module split. The ViT-kept split is a starting point; measure quantizing more or fewer modules against your task, since the accuracy-critical modules differ by model.
Compare against INT4 (AWQ/GPTQ) on your own task. The report's preference for FP8 is for small VLMs on a joint speed-accuracy metric; a larger or more redundant model may tolerate INT4's heavier compression.

How to run it in production¶

Gate on downstream accuracy, not reconstruction error. The claim is task-accuracy parity (GSM8K/MMLU held); make the deployment gate a real eval on your task, because tensor RMSE will look fine for FP8 either way.
Store and version the per-block scales with the checkpoint. They are part of the model; a scale/format mismatch at load time silently corrupts outputs.
Watch for hardware-dependent numerics. FP8 accumulation and scaling can differ across GPU generations and kernels; re-validate accuracy when you change the serving stack, not just the weights.
Keep a high-precision fallback. For the rare accuracy-critical request path, keep the option to serve the unquantized (or ViT-high-precision) model; PTQ is a deployment optimization, not a one-way door.

How to maintain it¶

Re-run the core-math assertions on any quantizer change. The E4M3 grid, the zero-block guard, and the FP8-vs-INT8 comparison are library-independent and catch a broken scale or grid immediately.
Re-calibrate after a data-distribution shift. Per-block scales are set from calibration magnitudes; a changed input distribution can push activations/weights outside the calibrated range.
Re-benchmark against newer methods periodically. Quantization moves fast; the AWQ/GPTQ/FP8-dynamic comparison the report ran is a snapshot, so re-check the field before locking a format for a long-lived deployment.
Do not silently widen the quantized set. Adding modules (especially the ViT) to FP8 to chase more compression can regress accuracy; treat any change to the module split as a change requiring a fresh eval.

Results¶

Reported by Guo et al. (2026), Qwen2.5-VL-3B, GSM8K and MMLU:

Method	Compression	Accuracy vs baseline	Speed
Block-wise FP8 (128x128, LLM-only, PTQ)	36.6%	held on GSM8K and MMLU	>140% compute speedup
Per-tensor FP8	comparable	reported to hurt small-model accuracy	baseline FP8
AWQ, GPTQ-int4, LLM-Compressor FP8-dynamic	(4-bit for INT4)	below block-wise FP8 on the joint metric	slower on the joint metric

Block-wise FP8 was the report's best overall on the combined speed-and-accuracy axis for these small models.

Failure modes¶

Quantizing the vision tower by accident. Folding the ViT into FP8 to gain compression can degrade visual features and task accuracy; keep it excluded unless measured otherwise.
Attributing an FP8 win to block granularity. On FP8, per-tensor and block-wise round-trip errors are nearly identical; a measured downstream change from block granularity usually comes from GEMM behaviour, not reconstruction error.
No FP8 tensor cores on the target. Compression is real everywhere, but the speedup needs native FP8 matmul; on older hardware you pay conversion overhead for no throughput gain.
Stale calibration. Scales set on an old input distribution clip or under-utilize the range after a data shift, quietly costing accuracy.
Scale/format drift at load. A mismatch between stored per-block scales and the serving kernel's expectations corrupts outputs without an obvious error.

References¶

Guo et al., "Thousand-GPU Large-Scale Training and Optimization Recipe for AI-Native Cloud Embodied Intelligence Infrastructure" (JD Technology, 2026): https://arxiv.org/abs/2603.11101
Micikevicius et al., "FP8 Formats for Deep Learning" (E4M3/E5M2 definitions): https://arxiv.org/abs/2209.05433
DeepSeek-V3 (128x128 block-wise FP8 GEMM at scale): https://arxiv.org/abs/2412.19437
LLM Compressor (block-wise FP8 PTQ, code): https://github.com/vllm-project/llm-compressor
Qwen2.5-VL-3B-Instruct (the report's model): https://huggingface.co/Qwen/Qwen2.5-VL-3B-Instruct