Markdown

Speculators: the draft-model checkpoint format¶

Scope: the toolchain and checkpoint format layer of speculative decoding, not the algorithm. This page covers vllm-project/speculators: what a draft checkpoint contains (config.json plus one model.safetensors), which fields a serving engine reads, how EAGLE-3 and DFlash differ as configurations inside the same format (which target layers are consumed, chain versus block, what must match the target), and the drift failures that a draft/target pair produces. The accept/reject mathematics and losslessness proof live in speculative decoding; the walltime and cost model in speculative decoding economics; acceptance measurement across workloads in evaluating speculative decoding; DeepSeek's separate DSpark drafter in DSpark speculative decoding.

Evidence: the speculators repository was cloned and read at commit 0faffeb3bd547b4451a978d7aaf26a2f01b83d62 (2026-08-25); its two algorithm doc pages were fetched from docs.vllm.ai on 2026-08-26, and their prose matches docs/user_guide/algorithms/{eagle3,dflash}.md at that commit sentence for sentence (compared after normalising links and list markers, not byte for byte, since one side is rendered HTML). The vLLM loader was read from vllm-project/vllm main at 61d4f56635e0f2faae4bd60e6da88a53c9d3affb (2026-08-26); it is not pinned to a release. Five published checkpoints on Hugging Face were fetched for real: their config.json bodies and, for three of them, the safetensors tensor index (via HTTP range requests on the header). The numpy block is executed and asserted, and its checkpoint numbers are those fetched values. No GPU, no vLLM server, and no training run were involved: every throughput, acceptance, or speedup figure attributed to the project is the project's claim, not a measurement made here.

What it is¶

Speculators is a library and a checkpoint format for producing speculative-decoding draft models and shipping them into vLLM. The README states the purpose directly: "Speculators is a library for training speculative decoding draft models that deploy directly to LLM inference engines like vLLM."¹ The format is deliberately thin: a draft checkpoint is an ordinary Hugging Face model directory, and the entire speculative-decoding contract is carried by extra keys inside config.json.

Two keys make a directory a speculators checkpoint:

speculators_model_type: the algorithm discriminator (eagle3, dflash, dspark, peagle, mtp). SpeculatorModelConfig.from_pretrained raises NotImplementedError if it is absent, and from_dict raises ValueError; both refuse to guess.²
speculators_config: a nested object holding algorithm, proposal_methods, default_proposal_method, and verifier. The verifier block records name_or_path and architectures so the engine can load and validate the target the draft was built against.²

Everything else is either standard Transformers (architectures, torch_dtype, transformers_version) or algorithm-specific. The draft's own decoder hyperparameters live in a nested transformer_layer_config, which is a full PretrainedConfig in its own right.

Two checkpoint files carry the model: config.json and model.safetensors. The Hub listings for RedHatAI/Qwen3-8B-speculator.eagle3 and RedHatAI/gemma-4-31B-it-speculator.dflash contain nothing else load-bearing besides a small auto_map module (eagle3.py, config.py) and a README. There is no separate tokenizer, no separate weight index, and no sidecar metadata file: the vocabulary mapping tensors t2d and d2t are registered buffers inside the same safetensors file.³

Why use it¶

One format, five algorithms. EAGLE-3, P-EAGLE, DFlash, DSpark, and MTP all serialise through SpeculatorModelConfig, discriminated by speculators_model_type and resolved through a Pydantic class registry (auto_package = "speculators.models", schema_discriminator = "speculators_model_type").² Adding an algorithm means registering a config class and a model class, not changing the training script.²⁵
The engine reads the checkpoint, not a flag. vLLM's SpeculatorsConfig.build_vllm_speculative_config derives {"method": speculators_model_type, "num_speculative_tokens": proposal_methods[0].speculative_tokens} straight from config.json, so vllm serve RedHatAI/Qwen3-8B-speculator.eagle3 needs no speculative flags at all.²¹
Provenance is in the artifact. speculators_version is stamped into every config (0.2.0.dev11 on the Llama-3.1-8B and Qwen3-8B EAGLE-3 checkpoints, 0.5.0 on the Qwen3-8B DFlash one), which is the only reliable way to tell which defaults were in force when a checkpoint was written.³
Third-party checkpoints convert in. convert_model() ingests EAGLE-3 checkpoints from SafeAILab/EAGLE, native MTP layers, and z-lab DFlash checkpoints, emitting the speculators format.⁵

When to use it (and when not)¶

Use it when a draft model has to be handed to a serving team as an artifact: the verifier binding, draft vocabulary mapping, proposal length, and algorithm are all inside the file, so the deployment cannot silently pair the wrong draft with the wrong target.
Use it when a pre-trained speculator already exists for the target. The RedHatAI collection covers Llama 3.1/3.3, Qwen3, Llama-4-Maverick, gpt-oss, and gemma-4 verifiers; training from scratch requires extracting hidden states from the target with vLLM first, which is a multi-GPU data-generation job before any training starts.¹³¹⁶
Do not use it to decide whether to speculate. That is a latency and cost question answered in speculative decoding economics, and a workload question answered in evaluating speculative decoding.
Do not use it for n-gram or prompt-lookup drafting: those have no draft model and therefore no checkpoint (speculative decoding).
MTP is a different shape. The decision guide notes MTP "does not train from scratch" and is "only available for models with native MTP support", so it finetunes an existing head rather than producing an independent draft.¹⁵

Architecture¶

The draft is not a standalone language model. It consumes hidden states from named layers of the target, which means the target must be instrumented to emit them, in training and in serving alike. Training therefore runs a vLLM server of the target purely as a hidden-state source (scripts/launch_vllm.py), and serving re-establishes the same tap inside the engine.

flowchart TD
  T["Target model (verifier)"] --> LV["launch_vllm.py: tap aux layers<br/>default (2, L//2, L-3) + last layer"]
  LV --> HS["hs_i.safetensors hidden states<br/>(offline cache) or streamed (online)"]
  PD["prepare_data.py: tokenize,<br/>loss mask, token_freq.pt"] --> HS
  PD --> VM["t2d / d2t vocab mapping<br/>(draft_vocab_size most frequent tokens)"]
  HS --> TR["train.py: --speculator-type eagle3 | dflash | dspark | peagle | mtp"]
  VM --> TR
  EXT["External checkpoint<br/>(EAGLE-3, z-lab DFlash, native MTP)"] --> CV["convert_model()"]
  CV --> CK
  TR --> CK["config.json + model.safetensors<br/>speculators_model_type, speculators_config, transformer_layer_config"]
  CK --> VL["vLLM SpeculatorsConfig.from_pretrained"]
  VL --> SPEC["speculative_config: method + num_speculative_tokens"]
  VL --> ARCH["architectures rewritten to<br/>Eagle3LlamaForCausalLM / DFlashDraftModel"]
  SPEC --> SRV["vllm serve: draft proposes, target verifies"]
  ARCH --> SRV
  T --> SRV

What the config actually contains¶

The table below is read from the two checkpoints fetched on 2026-08-26, with the field definitions from src/speculators/models/{eagle3,dflash}/config.py at 0faffeb.

Field	`Qwen3-8B-speculator.eagle3`	`gemma-4-31B-it-speculator.dflash`	Meaning
`speculators_model_type`	`eagle3`	`dflash`	Registry discriminator; vLLM's `method`
`speculators_version`	`0.2.0.dev11`	`0.5.0.dev53`	Which library defaults applied
`draft_vocab_size`	`32000`	`32000`	Rows of `lm_head`; size of `d2t`
`transformer_layer_config.vocab_size`	`151936`	`262144`	Target vocabulary; size of `t2d`
`transformer_layer_config.hidden_size`	`4096`	`5376`	Must equal the target's hidden size
`transformer_layer_config.num_hidden_layers`	`1`	`5`	Draft depth (the draft's own choice)
`transformer_layer_config.model_type`	`llama`	`llama`	Decoder family used to build layers
aux layer ids	absent	`[1, 17, 29, 47, 58]`	Which target layers feed the FC projection
`norm_before_residual`	`true`	n/a	EAGLE-3 residual placement
`block_size`	n/a	`8`	DFlash block width
`sample_from_anchor`	n/a	absent (defaults `false`)	Anchor is a bonus token, so `block_size - 1` drafts
`proposal_methods[0]`	`greedy`, `speculative_tokens: 3`	`greedy`, `speculative_tokens: 8`	Proposal length the engine will use
`speculators_config.verifier`	`Qwen/Qwen3-8B`, `["Qwen3ForCausalLM"]`	`google/gemma-4-31B-it`, `[]`	Target binding

Only one proposal method exists in the library: GreedyTokenProposalConfig, with speculative_tokens (default 5), verifier_accept_k (default 1), and accept_tolerance (default 0.0).⁴ Both non-default knobs are explicitly lossy. verifier_accept_k: 5 accepts a draft token that merely lands in the verifier's top 5, and accept_tolerance bounds how far in log likelihood it may sit from the top token; the field description claims "Values up to 5 have shown to minimally impact accuracy", with no citation in the source.⁴ Every published checkpoint inspected here ships verifier_accept_k: 1, accept_tolerance: 0.0, which is the distribution-preserving setting.

EAGLE-3 and DFlash as configurations¶

Both algorithms sit in the same format and differ in three concrete places: how many target layers they consume, whether drafting is a chain or a block, and how the first draft layer is wired.

EAGLE-3 concatenates auxiliary hidden states from selected target layers, projects them through fc, and drafts autoregressively.¹³ The first decoder layer is patched so q_proj, k_proj, and v_proj take 2 * hidden_size inputs, because the layer receives cat([embeds, hidden]).¹⁰ Only llama and qwen3 decoder families are implemented; model_classes[tl_config.model_type] raises KeyError for anything else, and vLLM raises ValueError: Unsupported model_type ... for Eagle3 speculator for the same set.¹⁰²¹
DFlash predicts a whole block in one forward pass using a non-causal mask over verifier hidden states and mask-token embeddings.¹⁴ It always builds Qwen3-style decoder layers regardless of what transformer_layer_config.model_type says. The gemma DFlash checkpoint proves it: model_type is llama, yet the safetensors index contains layers.N.self_attn.{q,k}_norm.weight of shape [256], which is the Qwen3 per-head RMSNorm that Llama layers do not have.³ The training docs say the same: "DFlash always uses a Qwen3-style decoder regardless".¹⁷
What must match the target. scripts/train.py:create_transformer_layer_config copies vocab_size, hidden_size, num_attention_heads, num_key_value_heads, head_dim, max_position_embeddings, initializer_range, rms_norm_eps, and the rope parameters from the verifier config. Only num_hidden_layers, intermediate_size, the sliding-window layout, and the decoder family are the draft's own choices. The --draft-config documentation states the constraint bluntly: "The draft hidden_size must match the verifier (mismatch is not yet supported)."¹⁷
Which target layers. When aux ids are unset, both sides independently default to (2, num_layers // 2, num_layers - 3): resolve_target_layer_ids in speculators emits it with a warning, and vLLM's get_eagle3_default_aux_hidden_state_layers returns exactly the same tuple.¹⁹²² That agreement is why the Qwen3-8B EAGLE-3 checkpoint works despite recording no aux ids at all, and it is also the trap: a draft trained with custom --target-layer-ids whose checkpoint omits them will be served against the default layers with no error.
Layer-index skew. For dflash and dspark, vLLM rewrites the ids: "target_layer_ids": [i - 1 for i in aux_layer_ids], with the comment "DFlash configs use different indexing for the target layers, see #40727". EAGLE-3 ids pass through unchanged.²¹ The same integer therefore means different layers depending on speculators_model_type.

Validated: the format contract, vocabulary pruning, and the serve decision¶

The block below is executed and asserted. It encodes the shape rules read out of the two repositories, runs them against three real published checkpoints (one of which fails), then quantifies the one acceptance effect that is specific to this format: the pruned draft vocabulary carried by t2d/d2t. The general accept/reject proof is not repeated here; it lives in speculative decoding, and the walltime theorem in speculative decoding economics. Run: python3 speculators-draft-model-format.py.

# Runnable on system python3 (numpy). Validates the speculators checkpoint format itself:
# (1) the config.json <-> safetensors shape contract that decides whether a draft loads
#     against its target at all, exercised on three REAL published checkpoints;
# (2) what a pruned draft vocabulary (the t2d/d2t buffers) costs in acceptance, and what it
#     does NOT cost (correctness);
# (3) whether the checkpoint's declared speculative_tokens is worth serving.
# Checkpoint numbers are the values fetched on 2026-08-26 from each repo's config.json and
# the safetensors header (first 8 bytes = header length, then the JSON tensor index).
import numpy as np

# ---------------------------------------------------------------- 1. shape contract

CKPTS = {
    "RedHatAI/Qwen3-8B-speculator.eagle3": dict(
        model_type="eagle3", hidden=4096, verifier_vocab=151936, draft_vocab=32000,
        draft_layers=1, tlc_model_type="llama", aux_ids=None, spec_tokens=3,
        block_size=None, sample_from_anchor=None,
        shapes={"fc.weight": (4096, 12288), "lm_head.weight": (32000, 4096),
                "embed_tokens.weight": (151936, 4096), "t2d": (151936,), "d2t": (32000,),
                "layers.0.self_attn.q_proj.weight": (4096, 8192)}),
    "RedHatAI/Qwen3-8B-speculator.dflash": dict(
        model_type="dflash", hidden=4096, verifier_vocab=151936, draft_vocab=32000,
        draft_layers=5, tlc_model_type="qwen3", aux_ids=[2, 10, 18, 26, 34], spec_tokens=7,
        block_size=8, sample_from_anchor=None,
        shapes={"fc.weight": (4096, 20480), "lm_head.weight": (32000, 4096),
                "embed_tokens.weight": (151936, 4096), "t2d": (151936,), "d2t": (32000,),
                "layers.0.self_attn.q_proj.weight": (4096, 4096)}),
    "RedHatAI/gemma-4-31B-it-speculator.dflash": dict(
        model_type="dflash", hidden=5376, verifier_vocab=262144, draft_vocab=32000,
        draft_layers=5, tlc_model_type="llama", aux_ids=[1, 17, 29, 47, 58], spec_tokens=8,
        block_size=8, sample_from_anchor=None,
        shapes={"fc.weight": (5376, 26880), "lm_head.weight": (32000, 5376),
                "embed_tokens.weight": (262144, 5376), "t2d": (262144,), "d2t": (32000,),
                "layers.0.self_attn.q_proj.weight": (8192, 5376)}),
}


def audit(name, c):
    """Every rule below is read off speculators @0faffeb or vLLM @61d4f56, not invented."""
    bad = []
    s = c["shapes"]
    # models/eagle3/core.py: num_aux = len(eagle_aux_hidden_state_layer_ids) or 3;
    # fc = Linear(num_aux * hidden_size, hidden_size). vLLM llama_eagle3.py mirrors it.
    num_aux = len(c["aux_ids"]) if c["aux_ids"] else 3
    if s["fc.weight"][1] != num_aux * c["hidden"]:
        bad.append(f"fc in_features {s['fc.weight'][1]} != {num_aux} aux x {c['hidden']}")
    # model.py DraftVocabMixin._init_vocab: lm_head is hidden -> draft_vocab_size,
    # embed_tokens is verifier_vocab_size rows, t2d/d2t sized by the two vocabularies.
    if s["lm_head.weight"] != (c["draft_vocab"], c["hidden"]):
        bad.append(f"lm_head {s['lm_head.weight']} != {(c['draft_vocab'], c['hidden'])}")
    if s["embed_tokens.weight"][0] != c["verifier_vocab"]:
        bad.append("embed_tokens rows != verifier vocab_size")
    if s["t2d"][0] != c["verifier_vocab"] or s["d2t"][0] != c["draft_vocab"]:
        bad.append("t2d/d2t length does not match (verifier_vocab, draft_vocab)")
    # eagle3 patches q/k/v to take cat([embeds, hidden]) => 2x hidden in_features
    # (models/eagle3/model_definitions.py _patch_eagle3_projections). dflash does not.
    want = 2 * c["hidden"] if c["model_type"] == "eagle3" else c["hidden"]
    if s["layers.0.self_attn.q_proj.weight"][1] != want:
        bad.append(f"q_proj in_features {s['layers.0.self_attn.q_proj.weight'][1]} != {want}")
    # models/dflash/core.py:212 speculative_tokens = block_size - (0 if sample_from_anchor else 1)
    if c["model_type"] == "dflash":
        anchor = bool(c["sample_from_anchor"])  # config default is False
        want_tok = c["block_size"] if anchor else c["block_size"] - 1
        if c["spec_tokens"] != want_tok:
            bad.append(f"speculative_tokens {c['spec_tokens']} != block_size-{0 if anchor else 1}"
                       f" = {want_tok} (sample_from_anchor={anchor})")
    # vllm/.../speculators/algos.py update_eagle3 raises on any other draft model_type.
    if c["model_type"] == "eagle3" and c["tlc_model_type"] not in ("llama", "qwen3"):
        bad.append(f"vLLM cannot dispatch eagle3 draft model_type {c['tlc_model_type']}")
    if bad:
        raise ValueError(f"{name}: " + "; ".join(bad))
    return num_aux


assert audit("eagle3-qwen3", CKPTS["RedHatAI/Qwen3-8B-speculator.eagle3"]) == 3
assert audit("dflash-qwen3", CKPTS["RedHatAI/Qwen3-8B-speculator.dflash"]) == 5

# FAILURE CASE, on a real published checkpoint: gemma dflash declares 8 speculative tokens
# with block_size 8 and no sample_from_anchor key, which the current rule makes 7.
try:
    audit("dflash-gemma", CKPTS["RedHatAI/gemma-4-31B-it-speculator.dflash"])
    raise AssertionError("gemma dflash should not pass the audit")
except ValueError as e:
    assert "speculative_tokens 8 != block_size-1 = 7" in str(e), e
    gemma_defect = str(e)

# Shape drift is caught, not silently absorbed: swap in a 3-aux fc for a 5-aux config.
broken = {**CKPTS["RedHatAI/Qwen3-8B-speculator.dflash"]}
broken["shapes"] = {**broken["shapes"], "fc.weight": (4096, 12288)}
try:
    audit("aux-drift", broken)
    raise AssertionError("aux-count drift must be rejected")
except ValueError as e:
    assert "fc in_features 12288 != 5 aux x 4096" in str(e)

# ------------------------------------------------- 2. what draft-vocab pruning costs
# The format ships t2d (bool, verifier_vocab) and d2t (int64, draft_vocab): the draft can
# only ever propose the draft_vocab_size tokens t2d selects. Acceptance is bounded by the
# target mass inside that set; correctness is not.
rng = np.random.default_rng(7)
V, D = 2000, 1200
p = rng.random(V) + 0.02
p /= p.sum()
keep = np.zeros(V, bool)
keep[np.argsort(-p)[:D]] = True           # t2d: keep the D most frequent tokens
oov = p[~keep].sum()                       # target mass the draft can never propose
q = np.where(keep, p, 0.0)
q = q / q.sum()                            # best possible pruned draft: p restricted to t2d


def spec_step(p, q, n, rng):
    """One speculative position: propose from q, accept w.p. min(1, p/q), else residual."""
    x = rng.choice(len(q), size=n, p=q)
    acc = rng.random(n) < np.minimum(1.0, p[x] / np.maximum(q[x], 1e-300))
    out = x.copy()
    r = np.maximum(0.0, p - q)
    r = r / r.sum() if r.sum() > 0 else p
    nrej = int((~acc).sum())
    if nrej:
        out[~acc] = rng.choice(len(p), size=nrej, p=r)
    return out, acc


N = 400_000
commits, acc = spec_step(p, q, N, rng)
emp = np.bincount(commits, minlength=V) / N
ceiling = np.minimum(p, q).sum()
assert abs(ceiling - (1.0 - oov)) < 1e-12, (ceiling, 1 - oov)      # exact ceiling identity
assert abs(acc.mean() - ceiling) < 0.01, (acc.mean(), ceiling)     # attained by the ideal draft
assert np.max(np.abs(emp - p)) < 0.01                              # still lossless
# The tokens outside the draft vocab are still emitted, at their exact target rate.
assert abs(emp[~keep].sum() - oov) < 0.01, (emp[~keep].sum(), oov)
assert emp[~keep].sum() > 0.0

# FAILURE CASE: the "accept if the draft token is the target's argmax" shortcut. This is
# what verifier_accept_k=1 reads like if the engine compares top-1 instead of running the
# rejection rule; it is lossless only under greedy targets, and skews sampled output.
pn = np.array([0.5, 0.3, 0.2])
qn = np.array([0.4, 0.35, 0.25])
top = int(np.argmax(pn))
xs = rng.choice(3, size=N, p=qn)
naive = np.where(xs == top, xs, rng.choice(3, size=N, p=pn))
emp_naive = np.bincount(naive, minlength=3) / N
assert emp_naive[top] > pn[top] + 0.15, (emp_naive[top], pn[top])  # argmax over-represented
assert np.max(np.abs(emp_naive - pn)) > 0.15                       # not the target distribution
# The same p, q under the rejection rule are lossless, so the skew is the rule's, not the draft's.
cn, an = spec_step(pn, qn, N, rng)
assert np.max(np.abs(np.bincount(cn, minlength=3) / N - pn)) < 0.01

# --------------------------------------- 3. is the declared speculative_tokens worth it
# General theorem lives in the economics page; this instantiates it for one checkpoint.
# Qwen3-8B eagle3 draft: 399,523,840 compute-relevant params (all tensors except the frozen
# embed_tokens copy and t2d/d2t) against Qwen3-8B's 8,190,735,360; bandwidth-bound proxy only.
c_draft = 399_523_840 / 8_190_735_360
gamma = 3                                   # the checkpoint's speculative_tokens


def block_eff(a, g):
    return g + 1.0 if a == 1.0 else (1.0 - a ** (g + 1)) / (1.0 - a)


def speedup(a, g, c):
    return block_eff(a, g) / (1.0 + g * c)


def breakeven(g, c):
    lo, hi = 0.0, 1.0
    for _ in range(200):
        mid = (lo + hi) / 2
        lo, hi = (mid, hi) if speedup(mid, g, c) < 1.0 else (lo, mid)
    return (lo + hi) / 2


a_star = breakeven(gamma, c_draft)
assert abs(speedup(a_star, gamma, c_draft) - 1.0) < 1e-9          # crossover is exact
assert speedup(a_star - 0.02, gamma, c_draft) < 1.0               # below it, a net loss
assert speedup(a_star + 0.02, gamma, c_draft) > 1.0
assert abs(block_eff(0.0, gamma) - 1.0) < 1e-12                   # only the bonus token
assert abs(block_eff(1.0, gamma) - 4.0) < 1e-12
# A 22% out-of-draft-vocab workload caps per-position acceptance at 0.78 (evaluation page's
# multilingual figure). Cheap draft: still far above break-even. Heavy draft: not.
assert speedup(0.78, gamma, c_draft) > 2.4
a_star_heavy = breakeven(gamma, 0.30)
assert a_star_heavy > 0.5 and speedup(0.50, gamma, 0.30) < 1.0

print("1. shape audit: eagle3-qwen3 num_aux=3 OK, dflash-qwen3 num_aux=5 OK")
print(f"   rejected -> {gemma_defect}")
print(f"2. draft vocab {D}/{V}: out-of-draft target mass={oov:.4f}, "
      f"acceptance ceiling={ceiling:.4f}, measured={acc.mean():.4f}, "
      f"commit maxdev={np.max(np.abs(emp - p)):.4f}")
print(f"   naive argmax-match rule on p={pn.tolist()}: p[top]={pn[top]:.2f} served as "
      f"{emp_naive[top]:.4f}; rejection rule on the same pair stays within "
      f"{np.max(np.abs(np.bincount(cn, minlength=3) / N - pn)):.4f}")
print(f"3. gamma={gamma}, draft/target cost={c_draft:.4f} -> break-even a*={a_star:.4f}, "
      f"speedup at a=0.78 is {speedup(0.78, gamma, c_draft):.2f}x; "
      f"a heavier draft (c=0.30) needs a*={a_star_heavy:.4f}")

Executed output:

1. shape audit: eagle3-qwen3 num_aux=3 OK, dflash-qwen3 num_aux=5 OK
   rejected -> dflash-gemma: speculative_tokens 8 != block_size-1 = 7 (sample_from_anchor=False)
2. draft vocab 1200/2000: out-of-draft target mass=0.1708, acceptance ceiling=0.8292, measured=0.8298, commit maxdev=0.0002
   naive argmax-match rule on p=[0.5, 0.3, 0.2]: p[top]=0.50 served as 0.7005; rejection rule on the same pair stays within 0.0012
3. gamma=3, draft/target cost=0.0488 -> break-even a*=0.1279, speedup at a=0.78 is 2.50x; a heavier draft (c=0.30) needs a*=0.5090

Three operational readings. First, the config-to-weights contract is checkable offline, from config.json plus the safetensors header alone, without loading a single tensor: fc.weight alone reveals how many target layers a checkpoint expects. Second, draft-vocabulary pruning costs acceptance and nothing else: the out-of-draft tokens are still emitted at their exact target rate through the residual, so a 32k draft vocab against a 151936-token target is a throughput decision, not a quality one. The acceptance ceiling equals one minus the out-of-draft target mass exactly, which is why the workload matters more than the checkpoint: evaluating speculative decoding reports roughly 22% of multilingual target tokens landing outside a 32k pruned vocabulary while Math and Coding are barely affected. Third, break-even acceptance for this checkpoint is around 0.13, far below any plausible workload, so a cheap EAGLE-3 draft is nearly always worth serving on latency grounds; the same is not true of a draft one third the target's cost, which needs better than 0.51.

How to use it¶

Serving needs no speculative flags: vLLM reads the checkpoint. The short form loads the target named in speculators_config.verifier.name_or_path.²³

# Reference template (needs vLLM installed + a GPU). Not executed here.
# Short form: the checkpoint names its own verifier.
vllm serve RedHatAI/Qwen3-8B-speculator.eagle3

# Long form: pass the target first, override num_speculative_tokens, or pair the
# draft with a quantized target. Keys as documented at speculators 0faffeb.
vllm serve Qwen/Qwen3-8B \
  -tp 1 \
  --speculative-config '{
    "model": "RedHatAI/Qwen3-8B-speculator.eagle3",
    "num_speculative_tokens": 3,
    "method": "eagle3"
  }'

Converting a third-party checkpoint uses the Python API, which supports eagle3, mtp, and dflash.⁵

# Reference template (needs speculators + transformers installed). Not executed here.
from speculators.convert import convert_model

convert_model(
    model="z-lab/Qwen3-8B-DFlash-b16",
    verifier="Qwen/Qwen3-8B",
    algorithm="dflash",
    output_path="./converted",
)

The speculators convert CLI is not equivalent to that API. Its --algorithm option is click.Choice(["eagle", "eagle3", "mtp"]), so dflash is rejected at the CLI even though convert_model implements it; and eagle, which the CLI accepts and documents with two worked examples, has no branch in convert_model and falls through to raise ValueError(f"Unsupported algorithm: {algorithm}").⁶⁵ Prefer the Python API until that is reconciled.

Loading in Python resolves the subclass from speculators_model_type automatically, and SpeculatorModel.from_pretrained will auto-convert an external DFlash checkpoint if given a verifier=; any other unrecognised external format raises NotImplementedError.⁷⁵

How to develop with it¶

Training runs from scripts/train.py, single-GPU or under torchrun, with --fsdp-shard for parameter sharding.¹⁷ The load-bearing choices:

--speculator-type selects the algorithm (eagle3, dflash, dspark, peagle, mtp), and several defaults key off it inside TrainConfig._resolve_derived_defaults: draft_arch becomes llama for eagle3 and qwen3 otherwise, num_layers becomes 5 for the DFlash family and 1 otherwise, loss_fn becomes ce for dflash and kl_div otherwise, block_size becomes 16 for dflash and 8 for dspark, and per_position_loss_weight becomes dpace for dflash.¹⁸
Hidden states come from the target, not the draft. Run scripts/launch_vllm.py <target> first. If custom --target-layer-ids are passed there, the same ids minus the appended final layer must be passed to train.py, because launch_vllm.py appends num_hidden_layers when --include-last-layer is on (the default) and training takes only the auxiliary layers.²⁰
--on-missing generate --on-generate delete is online training; --on-missing raise with a populated --hidden-states-path is offline; --on-generate cache is the hybrid that generates in epoch one and reuses afterwards.¹⁷
--dry-run builds the speculator, initialises weights, writes a checkpoint, and exits before training, specifically so the artifact can be validated in vLLM before a full run is launched; the same directory feeds back through --from-pretrained.¹⁷
Adding an algorithm means a directory under src/speculators/models/, a config class decorated @SpeculatorModelConfig.register("myalgo"), a model class decorated with the matching @SpeculatorModel.register, training factory classmethods, and CLI arguments; the registry means the training script never needs to learn about the new type.²⁵

How to maintain it¶

Pin what wrote the checkpoint. speculators_version is stamped automatically from the installed package. Treat it as the key fact when a checkpoint behaves differently from a fresh one: 0.2.0.dev11 and 0.5.0 checkpoints were written under different algorithm defaults.³²
Re-derive defaults, do not remember them. _resolve_derived_defaults changed the DFlash out-of-the-box recipe to 5 layers, D-PACE, cross-entropy, and block_size=16 on the basis of issue #979, while DSpark kept block_size=8 because "that combination was never tested there".¹⁸ Any DFlash checkpoint older than that change was trained under a different recipe.
Record provenance around the run. speculators/provenance.py collects the command line, the git SHA and git diff HEAD of an editable install, and versions of speculators, vllm, transformers, torch, and compressed-tensors. Keep that alongside the checkpoint; the checkpoint itself records only the speculators version.²⁶
Re-check the checkpoint after every library bump, using the offline audit in the executed block. A config.json written by an older version can carry field values the current code would never produce.
Verify the verifier binding is populated. VerifierConfig.architectures exists "to validate architecture compatibility of different verifiers with the speculator", but both RedHatAI DFlash checkpoints inspected here ship "architectures": [], so the check has nothing to compare against.²³

How to run it in production¶

Measure acceptance on the real workload before committing. scripts/evaluate/evaluate.py sweep --target http://localhost:8000/v1 runs the nine RedHatAI/speculator_benchmarks subsets and writes perf_results.csv; evaluate.py throughput reports acceptance only. SPEED-Bench categories are also supported via --dataset speedbench/<config> --speedbench-data-dir.²⁴ Method selection and interpretation belong to evaluating speculative decoding.
Do not assume the number in config.json is optimal. num_speculative_tokens in the long-form --speculative-config overrides proposal_methods[0].speculative_tokens. The checkpoint's value is the training-time --ttt-steps for EAGLE-3 (default 3) or block_size - 1 for DFlash, neither of which is a serving measurement.¹¹¹²
Match sample_from_anchor between checkpoint and engine. The DFlash page is explicit: "When deploying to inference engines, ensure the engine's sample_from_anchor setting matches your model's config", because the flag changes both which slots were trained and how predictions are harvested.¹⁴
Do not serve --sliding-window-non-causal checkpoints yet. The training docs state "vLLM currently doesn't support these models".¹⁷
Pair the draft with a quantized target deliberately. The serving tutorial presents combining a quantized target with an unquantized speculator as a supported configuration, and the decision guide states the draft architecture is independent of the verifier architecture including quantized variants.²³¹⁵ Neither claim is accompanied by a measurement in the repository.
DFlash maturity. The DFlash page carries its own caveat: "DFlash is under active development. Not all hardware configurations have been validated yet".¹⁴

Failure modes¶

Aux-layer count drift. fc is Linear(num_aux * hidden_size, hidden_size), with num_aux derived from the recorded ids or defaulting to 3.¹¹ A checkpoint trained on five aux layers cannot load against a config claiming three; the executed block rejects exactly this.
Aux-layer identity drift, which is silent. Neither the count nor the shapes change if the same number of different layers is tapped. A checkpoint that omits its aux ids is served against (2, L//2, L-3) by vLLM regardless of what it was trained on, with no warning at serve time.²²
norm_before_residual default disagreement. The speculators field defaults to False; vLLM's update_eagle3 reads config_dict.get("norm_before_residual", True).⁸²¹ A checkpoint that omits the key is built one way by speculators and another by vLLM, with no shape change and therefore no error. All three RedHatAI EAGLE-3 checkpoints inspected here set it explicitly to true, so this bites hand-written or third-party configs, not the published ones.
sliding_window_non_causal default disagreement. Same pattern, opposite direction: the speculators field defaults to False, while vLLM computes "causal": not config_dict.get("sliding_window_non_causal", True), so an omitted key yields non-causal masking in the engine.⁹²¹
DFlash layer-index off-by-one. vLLM subtracts one from every id for dflash and dspark but not for eagle3.²¹ Copying aux ids between algorithms shifts the tap by one layer.
A published checkpoint that fails its own rule. RedHatAI/gemma-4-31B-it-speculator.dflash declares speculative_tokens: 8 with block_size: 8 and no sample_from_anchor, where the current code emits block_size - 1 = 7; its Qwen3-8B sibling correctly declares 7.³¹² Which value the engine honours was not tested here.
Unloaded weights are NaN, not zero. _init_vocab initialises lm_head, embed_tokens, and verifier_lm_head to NaN deliberately "so it's easy to detect if they're never loaded".⁷ A NaN-producing draft is a load-path bug, not a training bug.
t2d must be loaded before verifier weights. load_verifier_weights raises ValueError if t2d is unset while use_draft_vocab is true, because it slices the verifier LM head by that mask.⁷
Lossy proposal knobs shipped as config. verifier_accept_k > 1 or accept_tolerance > 0 stop preserving the target distribution; the executed block shows what a top-1-match rule does to a sampled output stream.⁴
CLI and API disagree on conversion. speculators convert --algorithm dflash is rejected by the CLI; --algorithm eagle is accepted and then raises ValueError: Unsupported algorithm: eagle.⁶⁵

References¶

speculators repository (read at 0faffeb3bd547b4451a978d7aaf26a2f01b83d62, 2026-08-25): https://github.com/vllm-project/speculators
speculators EAGLE-3 algorithm page (fetched 2026-08-26): https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/eagle3
speculators DFlash algorithm page (fetched 2026-08-26): https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/dflash
speculators algorithm decision guide: https://docs.vllm.ai/projects/speculators/en/latest/user_guide/algorithms/decision_guide
speculators train.py CLI reference: https://docs.vllm.ai/projects/speculators/en/latest/cli/train
speculators launch_vllm.py CLI reference: https://docs.vllm.ai/projects/speculators/en/latest/cli/launch_vllm
speculators "Serve in vLLM" tutorial: https://docs.vllm.ai/projects/speculators/en/latest/user_guide/tutorials/serve_vllm
vLLM speculators config loader (read at 61d4f56635e0f2faae4bd60e6da88a53c9d3affb, 2026-08-26): https://github.com/vllm-project/vllm/blob/main/vllm/transformers_utils/configs/speculators/algos.py
RedHatAI speculator model collection: https://huggingface.co/collections/RedHatAI/speculator-models
RedHatAI/Qwen3-8B-speculator.eagle3 (config.json and safetensors index fetched 2026-08-26): https://huggingface.co/RedHatAI/Qwen3-8B-speculator.eagle3
RedHatAI/gemma-4-31B-it-speculator.dflash: https://huggingface.co/RedHatAI/gemma-4-31B-it-speculator.dflash
EAGLE paper (cited by the speculators EAGLE-3 page): https://arxiv.org/abs/2401.15077
DFlash paper (cited by the speculators DFlash page): https://arxiv.org/abs/2602.06036

README.md at 0faffeb, Overview section: "Speculators is a library for training speculative decoding draft models that deploy directly to LLM inference engines like vLLM." ↩
src/speculators/config.py at 0faffeb: VerifierConfig (name_or_path, architectures, the latter described as used "to validate architecture compatibility of different verifiers with the speculator, if needed"), SpeculatorsConfig (algorithm, proposal_methods, default_proposal_method, verifier, with a validator requiring default_proposal_method to match one configured proposal_type), and SpeculatorModelConfig (speculators_model_type, speculators_version defaulting to version("speculators"), speculators_config; auto_package = "speculators.models", schema_discriminator = "speculators_model_type"; from_pretrained raises NotImplementedError and from_dict raises ValueError when speculators_model_type is missing). ↩↩↩↩↩
Fetched 2026-08-26. config.json via https://huggingface.co/<id>/raw/main/config.json for RedHatAI/Qwen3-8B-speculator.eagle3, RedHatAI/Llama-3.1-8B-Instruct-speculator.eagle3, RedHatAI/gpt-oss-20b-speculator.eagle3, RedHatAI/Qwen3-8B-speculator.dflash, RedHatAI/gemma-4-31B-it-speculator.dflash; safetensors tensor index via HTTP range requests on the first bytes of model.safetensors for the first, fourth, and fifth. File listings via https://huggingface.co/api/models/<id>. The gemma DFlash index contains layers.{0..4}.self_attn.q_norm.weight and k_norm.weight of shape [256] despite transformer_layer_config.model_type == "llama". ↩↩↩↩↩↩
src/speculators/proposals/greedy.py at 0faffeb: GreedyTokenProposalConfig with proposal_type: Literal["greedy"], speculative_tokens (default 5, ge=1), verifier_accept_k (default 1, ge=1), accept_tolerance (default 0.0, ge=0.0, description ends "Values up to 5 have shown to minimally impact accuracy."). It is the only TokenProposalConfig subclass exported by src/speculators/proposals/__init__.py. ↩↩↩
src/speculators/convert/entrypoints.py at 0faffeb: convert_model(model, verifier, algorithm: Literal["eagle3", "mtp", "dflash"], output_path="converted", validate_device=None, **kwargs) with branches for eagle3, mtp, dflash and a final raise ValueError(f"Unsupported algorithm: {algorithm}"). maybe_convert_external_checkpoint returns the path unchanged when speculators_model_type is present, and otherwise only recognises DFlash ("dflash_config" in config_dict or any("DFlash" in a for a in architectures)), raising NotImplementedError for anything else and ValueError if no verifier was supplied. ↩↩↩↩↩
src/speculators/__main__.py at 0faffeb: the convert command declares click_type=click.Choice(["eagle", "eagle3", "mtp"]) and its docstring gives two --algorithm eagle examples, while convert_model (see above) has no eagle branch and does have a dflash one. Reported here as a docs/code and CLI/API divergence found at this commit, not as a fixed issue. ↩↩
src/speculators/model.py at 0faffeb: DraftVocabMixin._init_vocab sets use_draft_vocab = draft_vocab_size != verifier_vocab_size, registers t2d (bool, verifier vocab) and d2t (long, draft vocab), and initialises lm_head, embed_tokens, verifier_lm_head with torch.nn.init.constant_(..., torch.nan) under the comment "Initialize weights to nan so it's easy to detect if they're never loaded". load_vocab_mappings validates t2d.shape[0] == verifier_vocab_size, t2d.sum() == draft_vocab_size, and d2t.shape[0] == draft_vocab_size. load_verifier_weights raises ValueError ("t2d tensor hasn't been set...") when use_draft_vocab is true and t2d is unset. SpeculatorModel.from_pretrained resolves the subclass via registered_model_class_from_config. ↩↩↩
src/speculators/models/eagle3/config.py at 0faffeb: Eagle3SpeculatorConfig fields draft_vocab_size (default 32000), norm_before_residual (default False), target_hidden_size, eagle_aux_hidden_state_layer_ids, norm_before_fc (default False), fc_norm (default False, mutually exclusive with norm_before_fc), norm_output (default False), embed_requires_grad (default False); architectures defaults to ["Eagle3Speculator"] and transformer_layer_config defaults to Qwen3Config. ↩
src/speculators/models/dflash/config.py at 0faffeb: DFlashSpeculatorConfig fields draft_vocab_size (default 32000), block_size (default 8), target_hidden_size, aux_hidden_state_layer_ids, mask_token_id, sliding_window_non_causal (default False), sample_from_anchor (default False); architectures defaults to ["DFlashSpeculator"]. Two internal inconsistencies at this commit: the class docstring says "vocabulary mapping between draft (64K) and target (128K)" while draft_vocab_size defaults to 32000, and the registered model class is DFlashDraftModel (src/speculators/models/dflash/core.py:38), which is also what the published checkpoints declare, not DFlashSpeculator. ↩
src/speculators/models/eagle3/model_definitions.py at 0faffeb: Eagle3FirstLayerMixin._patch_eagle3_projections replaces q_proj, k_proj, v_proj with 2 * config.hidden_size input features, and forward splits the incoming cat([embeds, hidden]) at the midpoint. model_classes has exactly two entries, "llama" and "qwen3"; src/speculators/models/eagle3/core.py:65 indexes it with tl_config.model_type. ↩↩
src/speculators/models/eagle3/core.py at 0faffeb: num_aux = len(config.eagle_aux_hidden_state_layer_ids) if config.eagle_aux_hidden_state_layer_ids else 3; self.fc = torch.nn.Linear(num_aux * self.hidden_size, self.hidden_size, bias=False). Eagle3DraftModel.from_training_args builds GreedyTokenProposalConfig(speculative_tokens=kwargs["ttt_steps"]) and sets transformer_layer_config from the verifier config. ↩↩
src/speculators/models/dflash/core.py:212 at 0faffeb: speculative_tokens = block_size if sample_from_anchor else block_size - 1, feeding GreedyTokenProposalConfig(speculative_tokens=speculative_tokens). The same method sets "transformer_layer_config": verifier_config and takes block_size from kwargs.get("block_size", 8). ↩↩
docs/user_guide/algorithms/eagle3.md at 0faffeb, identical to the fetched page: "The target model produces hidden states at selected layers, which are concatenated and projected through an FC layer alongside token embeddings. These pass through Llama-style decoder layers (default: 1) and an LM head to produce draft logits." Note the divergence: the doc says "Llama-style", the training default --draft-arch is llama for eagle3, but Eagle3SpeculatorConfig.transformer_layer_config defaults to Qwen3Config and the code supports both families. The page lists pretrained speculators for Qwen/Qwen3-8B, meta-llama/Llama-4-Maverick-17B-128E-Instruct, openai/gpt-oss-20b, and google/gemma-4-31B-it. ↩↩
docs/user_guide/algorithms/dflash.md at 0faffeb, identical to the fetched page: "This block-parallel approach can yield 2--3x larger speedups than Eagle-3 on synchronous requests" (a project claim, with no benchmark on the page and none reproduced here); "The draft model uses Qwen3-style transformer layers but can be paired with any supported verifier"; the sample_from_anchor semantics ("Produces block_size - 1 speculative tokens" when false, block_size when true); "When deploying to inference engines, ensure the engine's sample_from_anchor setting matches your model's config"; and "DFlash is under active development. Not all hardware configurations have been validated yet". Its pretrained table lists only google/gemma-4-31B-it, while docs/index.md at the same commit also links a RedHatAI/Qwen3-8B-speculator.dflash. ↩↩↩
docs/user_guide/algorithms/decision_guide.md at 0faffeb: five algorithms, "All are lossless"; the support table gives draft layers as Llama-style for Eagle-3 and P-EAGLE, Qwen3-style for DFlash and DSpark, native MTP layers for MTP; "Eagle-3, P-EAGLE, DFlash, and DSpark can be paired with any supported verifier model (including quantized variants)"; MTP "does not train from scratch". Note that docs/user_guide/getting_started.md at the same commit lists only Eagle-3 and DFlash under "Supported Algorithms". ↩↩
docs/user_guide/getting_started.md at 0faffeb: "Training requires internal hidden states from the target model, which are extracted by serving the target model with vLLM", with online, offline, and hybrid modes described. ↩
docs/cli/train.md at 0faffeb: --speculator-type options; --draft-arch default llama with "DFlash always uses a Qwen3-style decoder regardless"; --draft-config with "The draft hidden_size must match the verifier (mismatch is not yet supported)"; --dry-run; --on-missing / --on-generate; --ttt-steps default 3; --block-size default 16 for dflash and 8 for dspark; --sliding-window-non-causal with "Note: vLLM currently doesn't support these models". One doc/code contradiction found at this commit: the page gives --max-anchors default 3072, while src/speculators/train/config/schema.py:229 sets default=512 (the DSpark model code separately uses 3072 as its own fallback in src/speculators/models/dspark/core.py). ↩↩↩↩↩↩
src/speculators/train/config/schema.py at 0faffeb, TrainConfig._resolve_derived_defaults: draft_arch -> llama if eagle3 else qwen3; norm_before_fc and norm_output -> is_eagle3; muon_lr -> 10 * lr; num_layers -> 5 for {dflash, dspark, dflash2} else 1; per_position_loss_weight -> dpace if dflash else fixed-exp-decay; loss_fn -> ce if dflash else kl_div; block_size -> 16 if dflash else 8. The docstring attributes the DFlash recipe to https://github.com/vllm-project/speculators/issues/979 and states DSpark "keeps block_size=8, since that combination was never tested there". norm_before_residual defaults to True in this training schema, against False in Eagle3SpeculatorConfig. ↩↩
src/speculators/models/utils.py at 0faffeb, resolve_target_layer_ids: returns the caller's ids when given, else [2, num_layers // 2, num_layers - 3] with a warning. ↩
docs/cli/launch_vllm.md at 0faffeb: --target-layer-ids default [2, num_layers//2, num_layers-3]; --include-last-layer default True appends num_hidden_layers; "If set, you must also pass the same layer ids to the training script using --target-layer-ids, excluding the final layer". ↩
vllm/transformers_utils/configs/speculators/algos.py in vllm-project/vllm main at 61d4f56 (file last modified by 2f55ef254c70e110d637beeedf48238977ebb683, 2026-08-22), fetched 2026-08-26. update_eagle3: pre_trained_config["norm_before_residual"] = config_dict.get("norm_before_residual", True); eagle3_arch_map = {"qwen3": "Eagle3Qwen3ForCausalLM", "llama": "Eagle3LlamaForCausalLM"} with raise ValueError(f"Unsupported model_type {model_type} for Eagle3 speculator") otherwise; aux ids copied only if config_dict.get("eagle_aux_hidden_state_layer_ids"). update_dflash: pre_trained_config["architectures"] = ["DFlashDraftModel"]; "target_layer_ids": [i - 1 for i in aux_layer_ids] under the comment "DFlash configs use different indexing for the target layers, see #40727"; "causal": not config_dict.get("sliding_window_non_causal", True). base.py build_vllm_speculative_config returns {"method": config_dict.get("speculators_model_type"), "num_speculative_tokens": proposal_methods[0]["speculative_tokens"]}, remapping peagle to eagle3 with parallel_drafting: True. ↩↩↩↩↩↩
vllm/model_executor/models/interfaces.py in vllm-project/vllm main at 61d4f56, fetched 2026-08-26: get_eagle3_default_aux_hidden_state_layers returns (2, num_layers // 2, num_layers - 3). vllm/model_executor/models/llama_eagle3.py at the same commit: self.num_aux_hidden_states = len(layer_ids) if layer_ids else 3 and self.fc_input_size = target_hidden_size * self.num_aux_hidden_states. ↩↩
docs/user_guide/tutorials/serve_vllm.md at 0faffeb: the short vllm serve ./checkpoints/checkpoint_best form ("vLLM automatically recognizes the speculators_config in your model and enables speculative decoding"), and the long form with --speculative-config '{"model": ..., "num_speculative_tokens": 3, "method": "eagle3"}', described as the way "to use a different target model with the speculator ... combine a quantized target model, with a speculative decoding model". ↩↩
docs/user_guide/tutorials/evaluating_performance.md at 0faffeb: python evaluate.py sweep --target http://localhost:8000/v1 over "all 9 subsets from RedHatAI/speculator_benchmarks" producing perf_results.csv; evaluate.py throughput for acceptance rates only; SPEED-Bench configs qualitative, throughput_1k, throughput_2k, throughput_8k, throughput_32k via --speedbench-data-dir. ↩
docs/developer/add_algorithm.md at 0faffeb: create a module under src/speculators/models, a config class with @SpeculatorModelConfig.register("myalgo"), a model class with the matching register decorator, training factory classmethods, and CLI arguments; "the training script doesn't need to know about every algorithm and adding a new algorithm doesn't require modifying the training script". ↩↩
src/speculators/provenance.py at 0faffeb: TRACKED_PACKAGES = ("speculators", "vllm", "transformers", "torch", "compressed-tensors"), plus git_sha, git_diff, and atomic_write helpers, described as "Used by training, evaluation, and vLLM-launch scripts to record command lines, git state, and package versions." ↩