Markdown

DFloat11: lossless BFloat16 compression¶

Scope: entropy-coding the BFloat16 exponent field so an LLM occupies roughly 70% of its BF16 size while producing bit-for-bit identical outputs, and the GPU kernel that decompresses weights inline on every forward pass. Covers the compression format, the hierarchical lookup tables and the two-phase decode kernel, the dfloat11 package, and the central trade this buys: capacity at the cost of speed. Proposed in "70% Size, 100% Accuracy" (Zhang et al., NeurIPS 2025).¹ Contrast with quantization, which is lossy and much faster.

The numpy block below is self-contained, was executed with system python3, and asserts every claim it prints, including the adversarial and failure cases. The dfloat11 snippets are reference templates taken from the upstream README; they need a GPU and are not executed here.

What it is¶

DFloat11 (DF11) is a lossless compression format for BF16 model weights, plus a CUDA kernel that decompresses them on the GPU while the model runs. It is not quantization. The decompressed weights are the same bits that went in, so the model's outputs are unchanged, not approximately unchanged.¹

The idea rests on one measurement. A BF16 number is 1 sign bit, 8 exponent bits, and 7 mantissa bits. Across the linear projections of Llama 3.1 8B, Gemma 2 9B, Qwen 2.5 14B, Mistral Small 24B and Llama 3.3 70B, the paper measures the Shannon entropy of each field: the sign carries a full 1.0 bits, the mantissa carries about 6.9 to 7.0 bits of its 7, and the exponent carries about 2.6 bits of its 8.¹ Trained weights cluster tightly around zero, so only about 40 of the 256 possible exponent values ever appear, and a handful of those dominate. The sign and mantissa look like noise and cannot be compressed; the exponent is a nearly-degenerate distribution stored in a fixed-width field.

So DF11 Huffman-codes the exponent byte and leaves everything else alone. Each weight becomes a variable-length exponent code plus a raw byte holding the sign and mantissa. The result averages 10.8 to 11.1 bits per weight, which is where the name comes from, and lands the checkpoint at 67.6% to 69.5% of BF16.¹

Model	BF16	DF11	Ratio	Avg bits
Llama 3.1 8B Instruct	16.06 GB	10.90 GB	67.84%	10.85
Llama 3.3 70B Instruct	141.11 GB	95.40 GB	67.61%	10.82
Llama 3.1 405B Instruct	811.71 GB	551.22 GB	67.91%	10.87
QwQ 32B	65.53 GB	44.65 GB	68.14%	10.90
Mistral Small 3	47.14 GB	31.86 GB	67.58%	10.81
FLUX.1 dev	23.80 GB	16.33 GB	68.61%	10.98
Stable Diffusion 3.5 Large	16.29 GB	11.33 GB	69.52%	11.12

Figures are from Table 1 of arXiv v3.¹ Version drift is worth knowing about if you cite this: for the four LLMs v1 also evaluated, v1's table reported worse ratios than v3's except for the 405B, whose row is unchanged (Llama 3.1 8B went from 69.98% and 11.20 bits in v1 to 67.84% and 10.85 in v3); FLUX.1 dev and Stable Diffusion 3.5 Large are not "worsened" in v1, they simply are not in it, since v1 never evaluated diffusion models at all. v1 carried Gemma rows that v3 silently drops. Gemma compresses worst of any model in either table, at 71.81% and 11.49 bits for gemma-2-9b-it.² Quote v3 for Llama, Qwen and Mistral, quote v1 for Gemma, and do not mix the two.

Why use it¶

One reason, and it is a good one: a model that does not fit, fits. Cutting 32% off the weights moves a checkpoint across a GPU memory boundary without touching its outputs, so no evaluation has to be re-run and no accuracy regression has to be argued about. Llama 3.1 405B is 811.71 GB in BF16, which does not fit an 8x80GB node; at 551.22 GB it does.¹

The alternative when a model overflows is to offload layers to CPU and pay PCIe on every token. Against that baseline (Hugging Face Accelerate offloading), DF11 is 2.31x to 46.24x faster.¹ The spread is not a property of DF11; it is a measure of how much of the model spilled to the host. The 46.24x is Mistral Small 3 (48 GB) on a 40 GB A100 at batch 1, where the BF16 baseline is crawling at 11.56 s/token. Within a fixed memory budget, the freed space also buys KV cache: 5.70x to 14.86x more generated tokens before OOM, measured at batch 1.¹

Against quantization, the argument is narrower and it is about risk, not speed. INT4 or FP8 shrinks the same model by 4x or 2x rather than 1.45x, and runs faster rather than slower. What DF11 offers is that you never have to prove the compressed model is still good, because it is the same model. The paper's own quantization datapoint is a fair statement of the risk it avoids: Llama 3.1 8B at INT8 loses 4.0 points on MATH Hard and 1.12 on GPQA CoT.¹ If your workload can absorb that, quantize; DF11 is for when it cannot.

When to use it (and when not)¶

Use it when the model sits just over a memory line and the next GPU is not available or not worth its price; when a lossy format is unacceptable for contractual, regulatory or evaluation-churn reasons; when the workload is a diffusion model, where the compute per forward pass is large enough to hide the decode (Stable Diffusion 3.5 pays +4.1% latency, FLUX.1 dev +5.5%, for about 28% less peak memory);¹ or when the honest comparison is against CPU offloading rather than against a model that fits.

Do not use it when the model already fits. This is the fact the abstract's "negligible overhead" framing obscures, and it is the single most important thing on this page. Appendix G of the paper measures DF11 against BF16 on the same GPUs with both models resident, and DF11 loses every time:¹

Model / GPU	Batch	BF16 tok/s	DF11 tok/s	DF11 slowdown
Llama 3.1 8B / A100-40GB	1	16.6	9.4	1.77x
Llama 3.1 8B / A100-40GB	128	2582.5	1146.3	2.25x
Llama 3.1 8B / A100-40GB	512	4422.2	3190.1	1.39x
Qwen 3 14B / A100-40GB	32	443.7	180.9	2.45x
Llama 3.3 70B / 4xA100-40GB	1	5.4	1.5	3.60x
Llama 3.3 70B / 4xA100-40GB	8	53.5	12.2	4.39x
Llama 3.3 70B / 4xA100-40GB	128	265.8	154.7	1.72x

The slowdown ratios are this page's arithmetic over the paper's throughput figures. The upstream README states the same thing in one line: "At batch size = 1, inference is approximately 2× slower than the original BF16 model."³ A user reproduced 31.91 tok/s BF16 against 9.51 tok/s DF11 on Qwen2.5-14B, and the maintainer confirmed the behaviour is expected.⁶

Also skip it when the checkpoint is already quantized. DF11 compresses the exponent field and nothing else, so an INT8 or INT4 model has no exponent redundancy left to harvest. The maintainer declined to support this for exactly that reason, noting that an INT8 model "might already use more than 7 bits of entropy."⁹ The code block below reproduces that argument from first principles.

Finally, do not reach for it if you need LoRA, fine-tuning, or tensor parallelism. None are supported (see Failure modes).

Architecture¶

flowchart TB
  subgraph OFF["Offline, once, CPU only"]
    A["BF16 checkpoint"] --> B["Split each weight:<br/>1 sign + 8 exponent + 7 mantissa"]
    B --> C["Huffman-code the exponent<br/>(~2.6 bits of entropy)"]
    B --> D["PackedSignMantissa<br/>1 raw byte per weight"]
    C --> E["EncodedExponent bitstream<br/>+ Gaps + BlockOutputPos"]
  end
  subgraph GPU["Online, every forward pass"]
    E --> F["DF11 weights resident in HBM"]
    D --> F
    F --> G["Per transformer block:<br/>one batched decode kernel"]
    G --> H["Phase 1: count elements,<br/>no HBM writes, prefix-sum"]
    H --> I["Phase 2: re-decode,<br/>reassemble BF16, coalesced write"]
    I --> J["BF16 weights in buffer"]
    J --> K["Block matmuls"]
    K --> L["Discard buffer, next block"]
  end

Three design choices carry the kernel, and each exists to work around a way that Huffman decoding is hostile to a GPU.

Hierarchical lookup tables. Decoding a Huffman code by walking the tree bit by bit is a serial pointer chase with a data-dependent branch at every step, which is close to a worst case for a GPU. The standard fix is a lookup table indexed by the next L bits, where L is the longest code, turning the walk into one load. But for LLM exponent trees L runs 24 to 32 bits, so that table wants up to 2^32 entries, about 4.29 billion.¹² Shared memory is around 100 KB per block by the paper's own working figure, and 228 KB per SM on an H100.¹⁶ So DF11 splits the tree into subtrees of height 8 and gives each one a 256-entry, one-byte-per-entry table. Four to eight such tables cover a real model, and the whole structure fits in at most (8+1) x 256 = 2304 bytes of SRAM.¹ The pointers between tables are free: exponent values 240 to 255 never occur in real weights (they would encode magnitudes around 2^113 and up), so those slots are repurposed as "descend into table n" markers.¹

A two-phase kernel. Variable-length codes mean a thread cannot know where its output belongs until every thread before it has finished decoding, which is a write-position dependency across the whole stream. DF11 breaks it by decoding twice. Each thread takes a contiguous 8 bytes of the encoded stream; in phase 1 every thread decodes its bytes and only counts how many weights it produced, writing nothing; a block-wide Blelloch prefix sum then converts those counts into output offsets; in phase 2 every thread decodes the same bytes again and writes to the offset it now knows.¹ Two small auxiliary arrays make this work: Gaps holds, for each thread, the bit offset of the first code that actually starts inside its 8-byte slice (a value in [0,31], so 5 bits), and BlockOutputPos holds one 32-bit integer per block rather than per thread, since per-thread offsets would eat the compression gain.¹

Block-level batching. A single weight matrix is too small to saturate a GPU, so decompression throughput climbs with matrix size. DF11 therefore decompresses all the matrices of one transformer block in a single kernel launch, issued just before that block runs.¹ Crucially, the decompressed BF16 weights are discarded right after the block's matmuls, so every weight is re-decompressed on every forward pass. There is no cache. This is the source of the entire latency cost.

How it works: the core algorithm¶

The block below builds DF11 end to end in numpy: split the BF16 fields, measure their entropy, Huffman-code the exponent, build both a flat and a hierarchical decoder, and prove the roundtrip is bit-exact. It then attacks the method: the case where the compression evaporates, the hard floor, what a single corrupted bit does, and why a quantized checkpoint gains nothing.

# dfloat11_core.py -- executed, numpy only. BF16 = 1 sign | 8 exponent | 7 mantissa.
# DFloat11 Huffman-codes the exponent byte and stores sign+mantissa raw.
import heapq
import numpy as np

RNG = np.random.default_rng(0)


def to_bf16(x: np.ndarray) -> np.ndarray:
    """float32 -> bf16 bit patterns (uint16), round-to-nearest-even."""
    u = x.astype(np.float32).view(np.uint32)
    rounded = (u + 0x7FFF + ((u >> 16) & 1)) >> 16
    return rounded.astype(np.uint16)


def split_bf16(w: np.ndarray) -> tuple[np.ndarray, np.ndarray]:
    """bf16 -> (exponent byte, sign|mantissa byte)."""
    exponent = (w >> 7).astype(np.uint8)
    sign = ((w >> 15) & 1).astype(np.uint8)
    mantissa = (w & 0x7F).astype(np.uint8)
    return exponent, ((sign << 7) | mantissa).astype(np.uint8)


def join_bf16(exponent: np.ndarray, signmant: np.ndarray) -> np.ndarray:
    sign = (signmant >> 7).astype(np.uint16)
    mantissa = (signmant & 0x7F).astype(np.uint16)
    return ((sign << 15) | (exponent.astype(np.uint16) << 7) | mantissa).astype(np.uint16)


def entropy_bits(counts: np.ndarray) -> float:
    p = counts[counts > 0] / counts.sum()
    return float(-(p * np.log2(p)).sum())


def huffman_lengths(counts: np.ndarray) -> dict[int, int]:
    """Canonical Huffman code lengths per symbol."""
    live = [(int(c), [int(s)]) for s, c in enumerate(counts) if c > 0]
    if len(live) == 1:
        return {live[0][1][0]: 1}
    depth = {s: 0 for _, syms in live for s in syms}
    heap = [(c, i, syms) for i, (c, syms) in enumerate(live)]
    heapq.heapify(heap)
    tie = len(heap)
    while len(heap) > 1:
        c0, _, s0 = heapq.heappop(heap)
        c1, _, s1 = heapq.heappop(heap)
        for s in s0 + s1:
            depth[s] += 1
        heapq.heappush(heap, (c0 + c1, tie, s0 + s1))
        tie += 1
    return depth


def canonical_codes(lengths: dict[int, int]) -> dict[int, tuple[int, int]]:
    codes: dict[int, tuple[int, int]] = {}
    code, prev_len = 0, 0
    for sym in sorted(lengths, key=lambda s: (lengths[s], s)):
        code <<= lengths[sym] - prev_len
        codes[sym] = (code, lengths[sym])
        code += 1
        prev_len = lengths[sym]
    return codes


def encode(exponent: np.ndarray, codes: dict[int, tuple[int, int]]) -> np.ndarray:
    bits: list[int] = []
    for s in exponent.tolist():
        code, ln = codes[s]
        bits.extend((code >> (ln - 1 - i)) & 1 for i in range(ln))
    return np.array(bits, dtype=np.uint8)


def build_lut(codes: dict[int, tuple[int, int]], max_len: int) -> tuple[np.ndarray, np.ndarray]:
    """Flat LUT: index by the next max_len bits -> (symbol, code length)."""
    sym = np.zeros(1 << max_len, dtype=np.uint8)
    ln = np.zeros(1 << max_len, dtype=np.uint8)
    for s, (code, cl) in codes.items():
        base = code << (max_len - cl)
        sym[base: base + (1 << (max_len - cl))] = s
        ln[base: base + (1 << (max_len - cl))] = cl
    return sym, ln


def build_hierarchy(codes: dict[int, tuple[int, int]], chunk: int = 8) -> list[tuple]:
    """Decompose one 2**max_len LUT into a tree of 2**chunk tables (the DF11 trick).

    Each table holds sym[256], len[256], next[256]. next >= 0 means "consume
    `chunk` bits and descend"; otherwise len > 0 marks a leaf. DF11 itself avoids
    the separate next array by reusing exponent values 240-255 as pointers.
    """
    tables: list[tuple[np.ndarray, np.ndarray, np.ndarray]] = []

    def make(sub: dict[int, tuple[int, int]]) -> int:
        idx = len(tables)
        sym = np.zeros(1 << chunk, dtype=np.uint8)
        ln = np.zeros(1 << chunk, dtype=np.uint8)
        nxt = np.full(1 << chunk, -1, dtype=np.int16)
        tables.append((sym, ln, nxt))
        deeper: dict[int, dict[int, tuple[int, int]]] = {}
        for s, (code, cl) in sub.items():
            if cl <= chunk:
                base = code << (chunk - cl)
                sym[base: base + (1 << (chunk - cl))] = s
                ln[base: base + (1 << (chunk - cl))] = cl
            else:
                prefix = code >> (cl - chunk)
                deeper.setdefault(prefix, {})[s] = (code & ((1 << (cl - chunk)) - 1), cl - chunk)
        for prefix, child in deeper.items():
            nxt[prefix] = make(child)
        return idx

    make(codes)
    return tables


def _window(padded: np.ndarray, pos: int, chunk: int) -> int:
    w = 0
    for b in padded[pos: pos + chunk]:
        w = (w << 1) | int(b)
    return w


def decode_hier(bits: np.ndarray, n: int, tables: list[tuple], chunk: int = 8) -> np.ndarray:
    """Decode by walking the LUT hierarchy, `chunk` bits per hop."""
    out = np.zeros(n, dtype=np.uint8)
    padded = np.concatenate([bits, np.zeros(64, dtype=np.uint8)])
    pos = 0
    for i in range(n):
        t, used = 0, 0
        while True:
            sym, ln, nxt = tables[t]
            w = _window(padded, pos + used, chunk)
            if nxt[w] >= 0:
                t, used = int(nxt[w]), used + chunk
                continue
            out[i] = sym[w]
            used += int(ln[w])
            break
        pos += used
    return out


def decode(bits: np.ndarray, n: int, sym: np.ndarray, ln: np.ndarray, max_len: int) -> np.ndarray:
    """Flat-LUT decode of n symbols starting at bit 0."""
    out = np.zeros(n, dtype=np.uint8)
    padded = np.concatenate([bits, np.zeros(max_len, dtype=np.uint8)])
    pos = 0
    for i in range(n):
        w = _window(padded, pos, max_len)
        out[i] = sym[w]
        pos += int(ln[w])
    return out


def stats(w_bf16: np.ndarray) -> tuple[float, float, float]:
    exponent, _ = split_bf16(w_bf16)
    counts = np.bincount(exponent, minlength=256)
    lengths = huffman_lengths(counts)
    avg = sum(counts[s] * l for s, l in lengths.items()) / counts.sum()
    return entropy_bits(counts), avg, 8.0 + avg


# LLM weights cluster near zero, so model them as normal. This is a synthetic
# stand-in for a real tensor, not a measurement of Llama's weights.
w = to_bf16(RNG.normal(0.0, 0.02, 200_000).astype(np.float32))
exponent, signmant = split_bf16(w)
counts = np.bincount(exponent, minlength=256)
H = entropy_bits(counts)
lengths = huffman_lengths(counts)
codes = canonical_codes(lengths)
avg_len = sum(counts[s] * l for s, l in lengths.items()) / counts.sum()
bits_per_weight = 8.0 + avg_len
MAXLEN = max(lengths.values())

print(f"distinct exponents used     : {int((counts > 0).sum())} of 256")
print(f"exponent entropy            : {H:.3f} bits")
print(f"huffman avg code length     : {avg_len:.3f} bits  (max {MAXLEN})")
print(f"sign+mantissa (stored raw)  : 8.000 bits")
print(f"bits per weight             : {bits_per_weight:.3f}  -> DFloat11")
print(f"compressed size             : {bits_per_weight / 16:.1%} of BF16")

# 1) Huffman lands within one bit of the Shannon limit.
assert H <= avg_len < H + 1.0

# 2) the other 8 bits are noise: their entropy is already ~8, so coding them is
#    wasted work. This is why DF11 leaves sign and mantissa raw.
h_raw = entropy_bits(np.bincount(signmant, minlength=256))
print(f"entropy of sign|mantissa    : {h_raw:.3f} bits (incompressible)")
assert h_raw > 7.9

# 3) lossless: the roundtrip returns the identical BF16 bit patterns.
bits = encode(exponent, codes)
sym_lut, len_lut = build_lut(codes, MAXLEN)
dec = decode(bits, exponent.size, sym_lut, len_lut, MAXLEN)
assert np.array_equal(dec, exponent)
rt = join_bf16(dec, signmant)
assert np.array_equal(rt, w)
print(f"roundtrip bit-exact         : {bool(np.array_equal(rt, w))} ({w.size} weights)")

# 4) the two-phase kernel's premise: given a per-chunk start offset, independent
#    workers decode disjoint chunks and land byte-identical to a serial decode.
CHUNK = 4096
offsets = np.concatenate([[0], np.cumsum([lengths[s] for s in exponent.tolist()])])
starts = offsets[::CHUNK][:-1] if exponent.size % CHUNK == 0 else offsets[::CHUNK]
parts = []
for i, start in enumerate(starts.tolist()):
    n = min(CHUNK, exponent.size - i * CHUNK)
    parts.append(decode(bits[start:], n, sym_lut, len_lut, MAXLEN))
parallel = np.concatenate(parts)
assert np.array_equal(parallel, exponent)
print(f"two-phase chunk decode == serial: {bool(np.array_equal(parallel, exponent))} "
      f"({len(starts)} chunks, block-offset array {len(starts) * 4} B)")

# 5) why one flat LUT will not do: it is indexed by the LONGEST code, so it costs
#    2**MAXLEN entries. The hierarchy of 8-bit tables decodes identically in a
#    fraction of the SRAM. Real models have 24-32 bit codes, far worse than this.
flat_kb = (sym_lut.nbytes + len_lut.nbytes) / 1024
hier = build_hierarchy(codes)
hier_kb = sum(t.nbytes for level in hier for t in level) / 1024
dec_h = decode_hier(bits, exponent.size, hier)
assert np.array_equal(dec_h, exponent)
print(f"flat LUT ({MAXLEN}-bit code)     : {flat_kb:.0f} KB "
      f"(H100 SM shared memory: 228 KB)")
print(f"hierarchical 8-bit LUTs     : {hier_kb:.2f} KB in {len(hier)} tables, "
      f"decode identical: {bool(np.array_equal(dec_h, exponent))}")
assert hier_kb < flat_kb / 20

# 6) adversarial: the win is a property of the weights, not of the code. On
#    uniform exponents Huffman degenerates to a fixed 8-bit code, the tensor does
#    not shrink at all, and the auxiliary array makes it a net loss. (This models
#    DF11's per-block BlockOutputPos, one uint32 per block; its per-thread Gaps
#    array costs a further 5 bits per thread.)
u = RNG.integers(0, 256, 200_000).astype(np.uint8)
u_counts = np.bincount(u, minlength=256)
u_len = sum(u_counts[s] * l for s, l in huffman_lengths(u_counts).items()) / u.size
overhead = 32.0 / CHUNK
print(f"uniform-exponent tensor     : {8.0 + u_len:.3f} bits/weight, "
      f"{8.0 + u_len + overhead:.3f} with the offset array (BF16 is 16.000)")
assert 8.0 + u_len == 16.0
assert 8.0 + u_len + overhead > 16.0

# 7) floor: even a tensor whose weights share one exponent pays 1 bit for it, so
#    9 bits per weight is the best DF11 can ever do.
d = np.full(1000, 0x3E, dtype=np.uint8)
d_counts = np.bincount(d, minlength=256)
d_len = sum(d_counts[s] * l for s, l in huffman_lengths(d_counts).items()) / d.size
print(f"single-exponent tensor      : {8.0 + d_len:.3f} bits/weight (hard floor)")
assert 8.0 + d_len == 9.0

# 8) corruption: the stream is not self-synchronising. One flipped bit desyncs the
#    decoder and silently rewrites most of the tensor. Checksum the artifact.
bad = bits.copy()
bad[5000] ^= 1
dec_bad = decode(bad, exponent.size, sym_lut, len_lut, MAXLEN)
wrong = int((dec_bad != exponent).sum())
print(f"1 flipped bit corrupts      : {wrong} of {exponent.size} weights")
assert wrong > 100
assert int(bits.sum()) != int(bad.sum())

# 9) DF11 exploits ONE redundancy, in the exponent. Round a tensor to 255 INT8
#    levels and the whole 16-bit word carries under 8 bits, yet DF11 still spends
#    ~10.4, because it never touches the mantissa. Ship that tensor as INT8.
q = to_bf16((RNG.integers(-127, 128, 200_000) * (0.02 / 127)).astype(np.float32))
_, _, q_bits = stats(q)
_, word_counts = np.unique(q, return_counts=True)
word_h = entropy_bits(word_counts)
print(f"int8-rounded tensor in bf16 : {q_bits:.3f} bits/weight from DFloat11, "
      f"but the word carries {word_h:.3f} bits")
assert q_bits > word_h + 2.0

Executed output:

distinct exponents used     : 20 of 256
exponent entropy            : 2.544 bits
huffman avg code length     : 2.590 bits  (max 16)
sign+mantissa (stored raw)  : 8.000 bits
bits per weight             : 10.590  -> DFloat11
compressed size             : 66.2% of BF16
entropy of sign|mantissa    : 7.971 bits (incompressible)
roundtrip bit-exact         : True (200000 weights)
two-phase chunk decode == serial: True (49 chunks, block-offset array 196 B)
flat LUT (16-bit code)     : 128 KB (H100 SM shared memory: 228 KB)
hierarchical 8-bit LUTs     : 3.00 KB in 3 tables, decode identical: True
uniform-exponent tensor     : 16.000 bits/weight, 16.008 with the offset array (BF16 is 16.000)
single-exponent tensor      : 9.000 bits/weight (hard floor)
1 flipped bit corrupts      : 156083 of 200000 weights
int8-rounded tensor in bf16 : 10.375 bits/weight from DFloat11, but the word carries 7.994 bits

Read the numbers carefully, because two of them are the whole method and two of them are its boundary.

The 2.544-bit exponent entropy against 7.971 bits for the sign-mantissa byte is the entire thesis, reproduced on a synthetic normal tensor: it recovers the paper's ~2.6-bit measurement without having its weights. The 10.590 bits per weight is slightly better than the paper's 10.81 to 11.12, which is expected: a pure normal is a touch more concentrated than a real tensor, and this model does not carry the auxiliary arrays. Treat this block as a proof of mechanism, not a substitute for the paper's measurements.

The uniform-exponent line is the adversarial case, and it fails more subtly than expected. Huffman does not expand a uniform distribution; it degenerates to a flat 8-bit code and returns exactly 16.000 bits per weight, at which point the auxiliary offset array tips it to a net loss. So a tensor with high exponent entropy does not blow up, it just quietly buys nothing while costing a decompression kernel. The int8-rounded line is the same trap wearing different clothes, and it independently reproduces the maintainer's stated reason for not supporting quantized models.⁹

The corruption line is an operational warning. A Huffman stream is not self-synchronising, so one flipped bit resynchronises the decoder into garbage and silently rewrote 156,083 of 200,000 weights here. Nothing raises. Checksum DF11 artifacts on download and after any copy.

How to use it¶

Install the package and load a pre-compressed checkpoint. The API wraps a Hugging Face model rather than replacing it, so generation is unchanged.

# Reference template from the upstream README (needs a CUDA GPU). Not executed here.
# pip install -U dfloat11[cuda12]
import torch
from dfloat11 import DFloat11Model
from transformers import AutoTokenizer

model_id = "DFloat11/Qwen3-8B-DF11"

model = DFloat11Model.from_pretrained(model_id, device_map="auto")

tokenizer = AutoTokenizer.from_pretrained(model_id)
tokenizer.pad_token = tokenizer.eos_token

prompt = "Question: What is a binary tree and its applications? Answer:"
inputs = tokenizer(prompt, return_tensors="pt", padding=True).to(model.device)

with torch.no_grad():
    output = model.generate(**inputs, max_new_tokens=256, do_sample=True)

print(tokenizer.batch_decode(output, skip_special_tokens=True))

The public surface is two names, DFloat11Model and compress_model. The installed package exposes no console-script entry point, so there is no dfloat11 command; the repo instead ships argparse scripts you run directly (inference.py for benchmarking, examples/compress_flux1/compress_flux.py for compression). Multi-GPU means Accelerate's device_map="auto" layer sharding, not tensor parallelism. The DFloat11 org on Hugging Face publishes 45 pre-compressed models, including Llama-3.1-405B-Instruct-DF11 at 547 GB, Qwen3-8B-DF11 at 11.17 GB, and FLUX.1-dev-DF11 at 16.33 GB.⁵

Check your CUDA version before anything else. The wheel ships a precompiled PTX file built against CUDA 12.2 (PTX ISA 8.2) and JIT-compiles it through the driver, so CUDA 12.0 and 12.1 fail with CUDA_ERROR_UNSUPPORTED_PTX_VERSION, and CUDA 13 is not supported upstream.¹⁰¹³ There is no GPU architecture floor: the PTX targets sm_52 and the kernel uses no arch-gated intrinsics. torch is not in install_requires and no version floor is documented anywhere, so pin it yourself.

How to develop with it: compressing your own checkpoint¶

compress_model compresses a BF16 checkpoint. Compression is CPU-only Huffman work; the GPU is touched only by the optional correctness check, so this runs on a machine without a big accelerator.

# Reference template from examples/compress_flux1/. Not executed here.
pip install -U diffusers dfloat11[cuda12]
python compress_flux.py \
    --model_name_or_path black-forest-labs/FLUX.1-dev \
    --save_path ./FLUX.1-dev-DF11 \
    --save_single_file \
    --check_correctness

# dfloat11/dfloat11.py:496 -- the real signature.
compress_model(model, pattern_dict: dict[str, list[str]], save_path: str,
               block_range: list[int] = [0, 10000],
               save_single_file: bool = True, check_correctness: bool = True)

The wall here is pattern_dict: a hand-written map from a regex over module paths to the list of linear submodules inside each block, and you must write one per architecture. The shipped example only covers FLUX; users report spending days adapting it to T5-XXL and Gemma-2.¹¹ Always leave check_correctness=True, which is the step that verifies the decompressed weights match the originals. compress_model asserts the input dtype is torch.bfloat16 and hard-fails on anything else.

Budget the time. The paper reports single-CPU-thread compression per transformer block of 191 s for Llama 3.1 8B, 547 s for Llama 3.3 70B, and 2133 s for Llama 3.1 405B.¹ Blocks are independent, so this parallelises across cores almost perfectly, and the repo ships a taskset-based parallel script. Compression is one-time; the artifact is then reusable forever.

How to run it in production¶

Size the memory honestly. Compressed weights are not the whole story: the kernel must materialise one transformer block's weights in BF16 before it can run the block's matmuls, and that buffer is real. The paper never quantifies it, which is a genuine gap; the only proxy it offers is Stable Diffusion 3.5, where DF11 peak memory is 11.78 GB against 11.33 GB of compressed weights.¹ So plan for compressed weights plus roughly one block of BF16 plus KV cache, and treat the 32% saving as an upper bound on what you actually get back.

Push batch size up. The decompression cost is constant per forward pass and independent of batch size,³ so it is pure overhead at batch 1 and amortises as the batch grows. That single fact should drive the whole deployment: DF11 is a poor fit for latency-sensitive, low-concurrency serving and a much better fit for throughput-oriented batch work. It is a worse fit than the numbers first suggest for the interactive case, because decode is exactly where batches are smallest.

Do not expect the rest of the serving stack to help. DF11 is not integrated into vLLM, SGLang, TensorRT-LLM, llama.cpp, or transformers proper; the vLLM feature request was closed as not_planned by a stale bot with no human reply.¹⁴ You get Hugging Face generate(), which means you also give up continuous batching, paged attention, and prefix caching. Weigh that against the memory saved, because a serving engine with paged KV may well recover more memory than DF11 does, and go faster.

Checksum every artifact. The corruption test above is not hypothetical: a single bit flip in the encoded stream silently corrupts most of a tensor and raises nothing.

How to maintain it¶

Watch the upstream, because it is close to dormant. LeanModels/DFloat11 last saw a commit on 2025-11-24, the PyPI package has sat at 0.5.0 since 2025-08-24, and the Hugging Face org has published nothing since 2025-09-30.⁴⁵ Twenty-two issues are open, including the CUDA 12.0/12.1 breakage and CUDA 13 support. Asked about serving frameworks in April 2025, the maintainer said DF11 "can definitely be made compatible" but that "it would require some work to adapt the model format and modify the code." That was a feasibility statement rather than a commitment, and nothing has shipped since.⁷

The live ecosystem has moved to a community fork, mingyi456/ComfyUI-DFloat11-Extended, which covers models the official org never did and adds LoRA support for a few architectures.¹⁵ Treat it with care: its own README reports that FLUX.2-klein outputs are "slightly different from BF16 output," which means losslessness, the entire reason to choose DF11, is currently broken there. If you use a fork, verify bit-exactness yourself on your model before trusting it.

Re-verify losslessness after any upgrade. It is cheap and it is the one property you are paying for: decompress, compare weight tensors bit-for-bit against the BF16 original, and run a fixed-seed generation against both.

Failure modes¶

Symptom	Cause	Response
Throughput 1.4x to 4.4x worse than BF16	The model fits in BF16, so you are paying decompression for nothing	Do not use DF11 when the model fits. It is a capacity tool, not a speed tool¹³
Slow at batch 1, fine at batch 128	Decompression cost is constant per forward pass	Raise batch size, or accept it. Do not use DF11 for low-latency single-stream decode³
`CUDA_ERROR_UNSUPPORTED_PTX_VERSION`	Driver older than CUDA 12.2; the shipped PTX is ISA 8.2	Upgrade to CUDA 12.2 or newer. CUDA 13 is not supported upstream¹⁰¹³
LoRA adapter fails to load, "no weight attribute"	DF11 deletes `module.weight` and rebinds it per forward	Not supported. Fine-tune in BF16, then compress the result⁸¹²
Compressing a quantized checkpoint saves almost nothing	The exponent redundancy DF11 harvests is already gone	Do not stack DF11 on INT8/INT4. Pick one⁹
Compression fails on a new architecture	`pattern_dict` is hand-written per model family	Write the regex map for your architecture; expect a slow first attempt¹¹
Outputs differ from BF16	Losslessness is broken, not "approximate"	Stop. Verify bit-exactness. A community fork ships this bug today¹⁵
Model silently degrades after a copy	One flipped bit desyncs the Huffman stream and rewrites most of a tensor	Checksum artifacts on download and after every copy
Memory saving smaller than 32%	The decompression buffer holds a block of BF16 weights	Budget compressed weights + one block of BF16 + KV cache¹

References¶

DFloat11 paper (v3, NeurIPS 2025): 70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float. https://arxiv.org/abs/2504.11651
DFloat11 paper v1 (carries the Gemma rows and kernel details dropped from v3). https://arxiv.org/abs/2504.11651v1
Upstream repository, LeanModels/DFloat11 (Apache-2.0). https://github.com/LeanModels/DFloat11
Pre-compressed models, Hugging Face DFloat11 org. https://huggingface.co/DFloat11
Community fork with wider model coverage, mingyi456/ComfyUI-DFloat11-Extended. https://github.com/mingyi456/ComfyUI-DFloat11-Extended
NVIDIA Hopper Tuning Guide (shared memory per SM). https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html
NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networks, a related weight-compression scheme "based on the entropy of floating-point numbers", with lossless and near-lossless modes. DFloat11 discusses it as related work and benchmarks against the nvCOMP ANS library. https://arxiv.org/abs/2410.20650

Zhang, Hariri, Zhong, Chaudhary, Sui, Hu, Shrivastava, 70% Size, 100% Accuracy (arXiv 2504.11651v3, 1 Jan 2026; NeurIPS 2025). Entropy of the BF16 fields and the ~40 occupied exponent values: §2.2 and Figure 1. Compression table: Table 1. Hierarchical LUTs, the 24-32 bit maximum code length, and the 2304-byte SRAM budget: §2.3.1. Two-phase kernel, Gaps, and BlockOutputPos: §2.3.2 and Algorithm 1. Block-level decompression and per-forward-pass discard: §2.3.3. CPU-offloading comparison (2.31-46.24x): Figure 4. Generation length (5.70-14.86x): Figure 5. BF16-fits throughput comparison: Appendix G, Figure 10; the slowdown ratios in this page's table are computed from those throughputs. Diffusion latency and memory: Table 3. Compression time per block: Appendix F, Table 5. INT8 accuracy drop: Appendix H. Hardware: Appendix E, Table 4 (A5000, A100-40GB, Quadro RTX 8000; no H100 and no 8x80GB node appears in any benchmarked experiment, so the 405B claim is arithmetic, 551.22 GB against 640 GB, and is not benchmarked in the paper. The word "H100" does appear once in the paper's introduction, as an illustrative example of a large-server capacity ("DGX A100/H100 with 8x80GB GPUs"), not tied to any measurement). ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩
arXiv 2504.11651v1 (15 Apr 2025), Table 2: gemma-2-9b-it 20.32 GB to 14.59 GB, 71.81%, 11.49 bits; gemma-3-12b-it 71.27%; gemma-3-27b-it 70.28%. v1 §3.3.1 also documents the maximum code length being forced to 32 by rebuilding the tree, and a LUT-pointer ambiguity case, both absent from v3. ↩↩
LeanModels/DFloat11 README, master branch, retrieved 2026-07-13. It states that decompression overhead is "constant" per forward pass and "independent of batch size" (the original sets those two spans in bold, dropped here), and, verbatim: "At batch size = 1, inference is approximately 2× slower than the original BF16 model, but the performance gap narrows significantly with larger batches." https://github.com/LeanModels/DFloat11 ↩↩↩↩
gh api repos/LeanModels/DFloat11, 2026-07-13: Apache-2.0, 641 stars, default branch master, last push 2025-11-24, 22 open issues. PyPI dfloat11 0.5.0, released 2025-08-24. ↩
https://huggingface.co/api/models?author=DFloat11, retrieved 2026-07-13: 45 models, newest Qwen-Image-Edit-2509-DF11 last modified 2025-09-30. Sizes summed from repo blobs. ↩↩
Issue #7, Qwen2.5-14B on an A40: 31.91 tok/s BF16 against 9.51 tok/s DF11; maintainer confirms the decompression overhead is expected. https://github.com/LeanModels/DFloat11/issues/7 ↩
Issue #8, framework integration, open since April 2025. https://github.com/LeanModels/DFloat11/issues/8 ↩
Issue #10, maintainer: "LoRA fine-tuning is not currently supported ... the full weight matrices are deleted." https://github.com/LeanModels/DFloat11/issues/10 ↩
Issue #15, maintainer on quantized models: "the gain is often minimal for quantized models ... an INT8 model might already use more than 7 bits of entropy." https://github.com/LeanModels/DFloat11/issues/15 ↩↩↩
Issue #19, CUDA_ERROR_UNSUPPORTED_PTX_VERSION on CUDA 12.0/12.1. https://github.com/LeanModels/DFloat11/issues/19 ↩↩
Issue #28, adapting pattern_dict to new architectures. https://github.com/LeanModels/DFloat11/issues/28 ↩↩
Issue #31, delattr(module, 'weight') breaks LoRA loading. https://github.com/LeanModels/DFloat11/issues/31 ↩
Issue #35, CUDA 13 unsupported. https://github.com/LeanModels/DFloat11/issues/35 ↩↩
vLLM issue #20003, "[Feature]: support dfloat11", opened 2025-06-24, closed 2025-10-23 as not_planned by the stale bot with no human reply. https://github.com/vllm-project/vllm/issues/20003 ↩
mingyi456/ComfyUI-DFloat11-Extended README, retrieved 2026-07-13, on FLUX.2-klein: "there is currently an unexplainable bug causing outputs to be slightly different from BF16 output." https://github.com/mingyi456/ComfyUI-DFloat11-Extended ↩↩
NVIDIA Hopper Tuning Guide: 228 KB of shared memory per SM on compute capability 9.0, with a 227 KB per-thread-block ceiling. https://docs.nvidia.com/cuda/hopper-tuning-guide/index.html ↩