Markdown

Speculative decoding economics: the walltime-cost model and when it stops paying¶

Scope: production economics for speculative decoding, separate from the accept/reject mechanics in speculative decoding. It uses Leviathan, Kalman, and Matias's low-batch walltime theorem for analytical screening, then requires direct latency, throughput, goodput, and cost measurements at the intended concurrency. A practitioner EAGLE3 case study is retained as an unreplicated warning, not as validation of a general performance model.

The EAGLE3-on-Llama-3.1-8B figures come from one practitioner report and were not reproduced for this page.⁴ Current vLLM behavior is cited from its documentation. The analytical code implements the published theorem only; it does not model high-batch verification cost.

What it is¶

Speculative decoding's accept rule gives the expected number of output tokens per target verification round, including the correction or bonus token: τ = (1 - α^(γ+1)) / (1 - α) for independent per-token acceptance α and draft depth γ.¹ It is not the expected number of accepted draft tokens. Theorem 3.8 gives the low-batch walltime improvement factor:

E[improvement factor] = (1 - α^(γ+1)) / ((1 - α)(γc + 1))

Here c is draft-step time divided by target-step time, γc prices the sequential draft steps, and 1 prices the target verification pass.¹ Estimate c from isolated step latency under the theorem's low-batch conditions. An inverse tokens-per-second ratio is only an approximation when both measurements perform comparable one-token decode steps.

The theorem is not a high-concurrency capacity model. vLLM documents that verification's effective batch becomes batch_size * K; beyond a hardware-specific critical batch, speculative decoding can worsen time per output token.² No single scalar interpolation is justified by these sources. Measure the complete server across offered load and use a batch-size schedule only after locating the crossover empirically.

Why use it¶

Separate acceptance from performance. τ can improve while latency or throughput regresses because drafting and verification consume time.
Screen drafter cost before a full deployment. The theorem exposes the low-batch crossover in c, but production approval still depends on an end-to-end load test.
Handle the high-batch regime explicitly. vLLM's dynamic schedule reduces K as concurrency rises because verification work scales with batch_size * K.²
Price measured capacity, not an analytical latency ratio. Cost per output token must use aggregate accepted-output throughput under the same offered load and SLO.

When to use it (and when not)¶

Reuse speculative decoding's general go/no-go rules; this page adds the cost-side checks specifically:

Use the theorem to reject a costly draft/target pair in low-batch decode before investing in a larger benchmark.
Use direct A/B load tests to compare TTFT, TPOT, request goodput, accepted output tokens per second, GPU memory, and power at the intended concurrency.
Use cost analysis when the drafter changes the GPU count, instance type, memory headroom, or power envelope.
Avoid static speculation depth beyond the measured high-batch crossover. Reduce K or disable speculation for that load range.
Do not transfer paper speedups. The EAGLE-3 paper's results and the practitioner regression use different protocols and do not predict a third deployment.³⁴

Architecture¶

flowchart TB
    M["Measure alpha and c<br/>under low-batch decode"] --> T["Screen with Theorem 3.8<br/>S = tau / (gamma*c + 1)"]
    T --> D{"Low-batch<br/>candidate?"}
    D -->|"no"| STOP["Do not enable speculation here,<br/>or pick a cheaper/better drafter"]
    D -->|"yes"| LOAD["A/B test complete server<br/>across offered load"]
    LOAD --> ECON["Price measured output throughput<br/>under the same SLO"]
    ECON --> G{"Net $/token<br/>improves?"}
    G -->|"no"| STOP
    G -->|"yes"| DEPLOY["Deploy behind a batch-size schedule<br/>(num_speculative_tokens_per_batch_size)"]
    DEPLOY --> MON["Monitor acceptance, TPOT, goodput,<br/>output throughput, and $/token"]

The analytical screen below implements Theorem 3.8 and checks it against Monte Carlo sampling. It makes no high-batch claim:

# Runnable on system python3 (numpy). Leviathan et al.'s Theorem 3.8
# walltime-improvement factor, cross-checked against Monte Carlo sampling.
import numpy as np


def tau(alpha, gamma):
    """Expected output tokens per verification round, including correction/bonus."""
    assert 0.0 <= alpha <= 1.0 and gamma >= 1
    if alpha == 1.0:
        return float(gamma + 1)
    return (1.0 - alpha ** (gamma + 1)) / (1.0 - alpha)


def walltime_speedup(alpha, gamma, c):
    """Theorem 3.8; valid only under the paper's cost assumptions."""
    assert c >= 0.0
    return tau(alpha, gamma) / (gamma * c + 1.0)


def simulate_walltime(alpha, gamma, c, n, rng):
    """Monte Carlo reference using the theorem's constant per-round cost."""
    accepts = rng.random((n, gamma)) < alpha
    leading = np.cumprod(accepts, axis=1).sum(axis=1)
    tokens_per_round = leading + 1
    sd_time = n * (gamma * c + 1.0)
    ar_time_for_same_tokens = tokens_per_round.sum()
    return ar_time_for_same_tokens / sd_time


rng = np.random.default_rng(3)

# 1. Monte Carlo converges to Theorem 3.8 for several parameter sets.
for alpha, gamma, c in [(0.7, 4, 0.1), (0.5, 3, 0.3), (0.9, 6, 0.05)]:
    mc = simulate_walltime(alpha, gamma, c, n=300_000, rng=rng)
    cf = walltime_speedup(alpha, gamma, c)
    assert abs(mc - cf) < 0.02, (alpha, gamma, c, mc, cf)

# 2. Boundaries include one correction token at alpha=0 and all draft tokens
# plus the bonus token at alpha=1.
assert tau(0.0, 4) == 1.0
assert tau(1.0, 4) == 5.0

# 3. The analytical break-even cost is c* = (tau - 1) / gamma.
alpha, gamma = 0.75, 4
c_star = (tau(alpha, gamma) - 1.0) / gamma
assert walltime_speedup(alpha, gamma, c_star - 1e-6) > 1.0
assert abs(walltime_speedup(alpha, gamma, c_star) - 1.0) < 1e-12
assert walltime_speedup(alpha, gamma, c_star + 1e-6) < 1.0

# 4. Raising drafter cost must strictly reduce the theorem's speedup.
cs = np.linspace(0.02, 1.0, 50)
speedups = np.array([walltime_speedup(alpha, gamma, c) for c in cs])
assert np.all(np.diff(speedups) < 0)

print("Theorem 3.8 walltime model: PASS")
print(f"  alpha=0.75, gamma=4 analytical c*={c_star:.6f}")

Executed output:

Theorem 3.8 walltime model: PASS
  alpha=0.75, gamma=4 analytical c*=0.512695

How to use it¶

Use a measured batch-size schedule. Current vLLM accepts inclusive [start_bs, end_bs, K] ranges in num_speculative_tokens_per_batch_size; K = 0 disables drafting in that range.² The following values are illustrative reference inputs, not recommendations:

# Reference template (needs vllm installed + a GPU). Not executed here.
from vllm import LLM

LLM(
    model="<target-model>",
    speculative_config={
        "method": "eagle3",
        "model": "<eagle3-head-repo-or-path>",
        "num_speculative_tokens": 5,
        # Illustrative only; cover the server's configured maximum batch size.
        # These ranges must come from an A/B load test.
        "num_speculative_tokens_per_batch_size": [[1, 32, 5], [33, 96, 2], [97, 512, 0]],
    },
)

Current vLLM documents dynamic scheduling as tested with EAGLE, EAGLE-3, and DFlash. It is not compatible with data parallelism: vLLM disables the dynamic schedule under DP and falls back to static num_speculative_tokens to avoid collective divergence.² Validate method, runner, parallelism, and version compatibility before rollout.

The $/token production decision. Use the measured aggregate accepted-output throughput multiplier M from an A/B load test at the same offered load and SLO. Do not substitute the theorem's single-stream walltime factor. Price any changed GPU count, instance type, or power allocation:

# Runnable on system python3 (numpy). Turns the cost formula
# (cost per 1M tokens = pod $/hr * 1e6 / (tokens/sec * 3600)) into a go/no-go
# decision using a measured accepted-output throughput multiplier M.
import numpy as np


def cost_per_million_tokens(pod_dollars_per_hr, tokens_per_sec):
    assert pod_dollars_per_hr > 0 and tokens_per_sec > 0
    return pod_dollars_per_hr * 1_000_000 / (tokens_per_sec * 3600)


def speculative_cost_per_million(pod_dollars_per_hr_baseline, tokens_per_sec_baseline,
                                  throughput_multiplier, extra_dollars_per_hr=0.0):
    """Price measured aggregate output throughput and changed hourly cost."""
    assert throughput_multiplier > 0
    new_rate = tokens_per_sec_baseline * throughput_multiplier
    new_cost_per_hr = pod_dollars_per_hr_baseline + extra_dollars_per_hr
    return cost_per_million_tokens(new_cost_per_hr, new_rate)


def breakeven_extra_dollars(pod_dollars_per_hr_baseline, throughput_multiplier):
    """Extra $/hr that exactly cancels measured throughput multiplier M."""
    return pod_dollars_per_hr_baseline * (throughput_multiplier - 1.0)


base = cost_per_million_tokens(pod_dollars_per_hr=2.00, tokens_per_sec=45.0)

# 1. Higher measured throughput (M>1) at the same hourly cost lowers $/token.
better = speculative_cost_per_million(2.00, 45.0, throughput_multiplier=1.8)
assert better < base

# 2. Lower accepted-output throughput makes $/token worse.
worse = speculative_cost_per_million(2.00, 45.0, throughput_multiplier=0.85)
assert worse > base

# 3. A throughput gain can still lose after pricing changed hardware.
nominal_win_but_pricier_box = speculative_cost_per_million(
    2.00, 45.0, throughput_multiplier=1.15, extra_dollars_per_hr=1.20)
assert nominal_win_but_pricier_box > base

# 4. The break-even boundary is exact.
be = breakeven_extra_dollars(2.00, throughput_multiplier=1.8)
at_breakeven = speculative_cost_per_million(2.00, 45.0, 1.8, be)
assert abs(at_breakeven - base) < 1e-9
assert speculative_cost_per_million(2.00, 45.0, 1.8, be * 0.5) < base
assert speculative_cost_per_million(2.00, 45.0, 1.8, be * 1.5) > base

# 5. Boundary: unchanged throughput and hourly cost reproduce the baseline.
assert abs(speculative_cost_per_million(2.00, 45.0, 1.0, 0.0) - base) < 1e-12

print("Dollar-per-token model: PASS")
print(f"  base=${base:.2f}/M tok; M=1.8x same-cost -> ${better:.2f}")
print(f"  M=0.85x -> ${worse:.2f}; M=1.15x plus $1.20/hr -> ${nominal_win_but_pricier_box:.2f}")
print(f"  break-even extra at M=1.8x: ${be:.2f}/hr")

Executed output:

Dollar-per-token model: PASS
  base=$12.35/M tok; M=1.8x same-cost -> $6.86
  M=0.85x -> $14.52; M=1.15x plus $1.20/hr -> $17.18
  break-even extra at M=1.8x: $1.60/hr

How to develop with it¶

Measure c from comparable decode steps. Use draft-step time divided by target-step time in the low-batch regime. The inverse tokens-per-second ratio is an approximation only when both tests perform comparable one-token steps.
Measure α per traffic domain, not as one global average. Acceptance rate is workload-dependent; evaluating speculative decoding (SPEED-Bench) covers the standardized methodology for measuring it across domains and input-length buckets, which feeds directly into this page's τ(α, γ) term.
Sweep offered load with speculation on and off. Record TTFT, TPOT, request goodput, aggregate accepted-output tokens per second, memory use, and power. Locate the crossover directly instead of fitting an unsupported scalar.

How to maintain it¶

Re-measure c and the compute-bound crossover on any GPU generation or model swap. Both the baseline single-stream tokens/sec and the memory-bandwidth-to-compute crossover point shift with hardware, so a schedule tuned on one GPU class does not transfer to another.
Re-run the break-even calculation when pricing or measured throughput changes. breakeven_extra_dollars depends on baseline hourly cost and the measured multiplier M.
Retain protocol metadata with every result. Model revisions, engine version, GPU, sampling parameters, prompt/output lengths, concurrency, and SLO define the measurement. The EAGLE-3 paper's reported range is not portable to a different protocol.³

How to run it in production¶

Gate speculation with a measured batch-size schedule, using num_speculative_tokens_per_batch_size to reduce or zero draft depth beyond the crossover.²
Track cost per million accepted output tokens with acceptance, TTFT, TPOT, request goodput, and aggregate output throughput. Compare routes at the same SLO and offered load.
Re-benchmark every target, drafter, GPU, engine, and traffic change. The practitioner regression is a warning that a supported configuration can lose, not a reusable threshold.⁴

Failure modes¶

Failure mode	Cause	Mitigation
Adequate-looking `τ`, net latency loss	`τ` ignores drafter time; Theorem 3.8 also does not model the high-batch server.	Measure `c` for low-batch screening, then A/B test the complete server.
Gain disappears as load rises	Verification's effective batch grows as `batch_size * K` and crosses the hardware-specific critical batch.²	Reduce or zero `K` above the measured crossover.
Published speedup fails to transfer	Model, engine, hardware, sampling, lengths, or concurrency differ.³	Reproduce the full protocol before capacity planning.
Throughput gain, cost regression	Added hourly cost exceeds `baseline_cost * (M - 1)`.	Price measured aggregate output throughput and the complete serving allocation.
Dynamic schedule silently becomes static	Data parallelism is enabled; current vLLM disables the schedule to prevent collective divergence.²	Use the documented static fallback under DP or validate a non-DP deployment.

References¶

Leviathan, Kalman, Matias, "Fast Inference from Transformers via Speculative Decoding," ICML 2023: https://arxiv.org/abs/2211.17192
Li et al., "EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Test," NeurIPS 2025: https://arxiv.org/abs/2503.01840
vLLM, Dynamic Speculative Decoding: https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/ and SpeculativeConfig: https://docs.vllm.ai/en/latest/api/vllm/config/speculative/
Vizuara, "Speculative Decoding: Theory and Implementation in vLLM" (the EAGLE3-on-Llama-3.1-8B case study, cost-per-token formula, and the τ ≳ 3 practitioner heuristic): https://vizuara.substack.com/p/speculative-decoding-theory-and-implementation
Speculative decoding (accept rule, losslessness, drafter families); Evaluating speculative decoding (SPEED-Bench) (measuring α across domains)

Leviathan, Kalman, Matias, "Fast Inference from Transformers via Speculative Decoding," ICML 2023, https://arxiv.org/abs/2211.17192. Theorem 3.8 defines the stated expected walltime factor, with c equal to draft-model run time divided by target-model run time and γ sequential draft completions per round. ↩↩
vLLM, Dynamic Speculative Decoding, https://docs.vllm.ai/en/latest/features/speculative_decoding/dynamic_speculative_decoding/. The documentation explains the batch_size * K verification cost, inclusive schedule ranges, K = 0, tested methods, runner constraints, and the data-parallel fallback to static num_speculative_tokens. ↩↩↩↩↩↩↩
Li et al., "EAGLE-3," NeurIPS 2025, https://arxiv.org/abs/2503.01840. Reports 4.1x to 6.5x speedup at temperature 0 across Vicuna-13B, Llama-3.1-8B, and Llama-3.3-70B under the paper's benchmark protocol. ↩↩↩
Vizuara, "Speculative Decoding: Theory and Implementation in vLLM," https://vizuara.substack.com/p/speculative-decoding-theory-and-implementation. Reports an EAGLE3 deployment on Llama-3.1-8B-Instruct using a single 48 GB GPU and 500 ShareGPT prompts. It reports acceptance length τ = 1.81, about 45 tok/s for the target in its single-stream test, and a result slower than the no-speculation baseline. This practitioner benchmark is neither independently reproduced nor peer reviewed. ↩↩↩