Skip to content
Markdown

FreeToken: bandwidth-adaptive edge MoE serving

Scope: FreeToken, an edge-native MoE serving engine (paper arXiv 2608.16157, code FlashML-org/FreeToken), and specifically the one decision that distinguishes it from every other expert-offload engine: given m experts that a decode step routed to but the GPU cache does not hold, how many to pull over PCIe and how many to execute in place on the CPU. This page covers the q* split and its measured inputs, the exact integer rounding the shipped CUDA kernel performs, full-layer double-buffered prefill, the shared cross-layer LRU expert cache, the tool-call state anchor, and the gap between what the paper claims and what the repository implements at the pinned commit. It does not re-teach MoE routing (see MoE routing and expert load balancing), the expert FFN kernels themselves (MoE expert backends and grouped-GEMM kernels), the sparse-scaling rationale (Mixture-of-experts: sparse scaling), or multi-GPU expert sharding (expert parallelism for MoE inference), which is the datacenter answer to the same memory problem. The nearest neighbours on the small-box axis are Colibri (GLM-5.2 on 25 GB of RAM) and DwarfStar (ds4), which both stream experts but always fetch them; FreeToken's contribution is that a missing expert can also be computed where it already lives.

The q* policy is reproduced here in numpy, byte-for-byte against the integer rule in the repository's own device kernel, and asserted: against the paper's Eq. 4, against a brute-force optimum, at both bandwidth endpoints, on all six machines of the paper's Table 1, and on two cases where the design's premises fail. Every throughput and latency figure quoted is the paper's own measurement, not reproduced here: this page's authoring host has no NVIDIA GPU, so FreeToken was never built or run. The repository was cloned and read at commit 9ef3651309fe4058672f2cc92069238dea06be1b (2026-08-26); the paper PDF was fetched from arXiv on 2026-08-26. Shell invocations are reference templates against that commit, unexecuted. Where the repository implements something narrower than the paper describes, both are stated with file and line pointers rather than resolved.

What it is

FreeToken is a serving engine for MoE models whose complete expert pool does not fit in GPU memory, targeted at a single consumer or workstation GPU rather than a cluster. Non-expert weights stay resident on the GPU; the complete routed-expert pool lives in host memory and "remains the source of truth"; whatever GPU memory is left becomes one elastic expert cache shared by every MoE layer, keyed on (layer, expert) identifiers rather than tensor shards.1

That much is the standard expert-offload architecture, and the paper says so: EdgeMoE, Mixtral-offloading, MoE-Infinity, ProMoE, ExpertFlow and FineMoE all "converged on the architecture FreeToken also adopts", differing only in "how well they predict misses, but not in how they serve them: every miss is ultimately a PCIe transfer".1 FreeToken's claim is on the second axis. Three mechanisms:

  • Bandwidth-adaptive execution (q*). Each decode step, at each MoE layer, the m missing experts are partitioned into a cache-fill set F (transferred over PCIe, executed on the GPU, left resident for reuse) and a CPU-execution set C (executed in place from the host-resident pool, residency unchanged). The two run concurrently and the partial sums merge exactly, "preserving the exact MoE output without algorithmic approximation". The split size q = |F| comes from two bandwidths profiled on the deployed machine.1
  • Semantic-aware caching. A shared cross-layer LRU expert cache whose contents follow the router rather than a placement frozen at load time, plus recurrent-state checkpoints anchored at agent-harness edit boundaries so a context edit re-prefills only the new suffix.1
  • Elastic memory. The GPU expert cache can be resized and rebuilt at a scheduler safe point under a revised VRAM budget without restarting the engine or reloading the host pool, because "GPU memory affects only performance, never correctness".1

The paper places it as following "the GPU-centric serving architecture established by systems such as SGLang and vLLM", combining paged KV cache management and radix-based prefix reuse with FlashInfer and Flash Linear Attention kernels. The README is more specific about lineage: under Acknowledgment it says FreeToken "was deeply inspired by mini-sglang", and "learned the design and reused code from" SGLang, vLLM, FlashInfer, flash-linear-attention, LightLLM and llama.cpp.12

Why use it

The reason a bandwidth-aware split exists at all is that on consumer hardware the two paths to a missing expert are within small integer factors of each other, and which one wins flips between machines. The paper's Table 1 gives both bandwidths measured "on the deployed tensor shapes rather than taken from platform specifications":1

System GPU (VRAM) PCIe B_P (GB/s) DRAM B_H (GB/s) B_H / B_P
5090 (server) RTX 5090 (32 GB) 5.0 x16 52.7 DDR5 77.3 1.47
4090 RTX 4090 (24 GB) 4.0 x16 25.1 DDR4 63.2 2.52
3090 RTX 3090 (24 GB) 4.0 x16 25.3 DDR4 56.7 2.24
5090 desktop RTX 5090 (32 GB) 5.0 x16 49.0 DDR5 53.8 1.10
4060 laptop RTX 4060 Laptop (8 GB) 4.0 x8 11.8 LPDDR5 47.5 4.03
PRO 6000 RTX PRO 6000 (96 GB) 5.0 x16 51.5 DDR5 178 3.46

B_P is the measured host-to-device expert-transfer bandwidth over PCIe; B_H is the measured effective bandwidth of the CPU-side MoE expert kernel. The ratio spans 1.10 to 4.03 across six machines. A 4060 laptop on a PCIe 4.0 x8 link should push almost every miss to the CPU; a 5090 desktop whose dual-channel DDR5 barely outruns its PCIe 5.0 link should push almost every miss over the wire. The paper's argument is that this ratio "cannot be read from specification sheets", so the division has to be made "quantitatively, on the machine it actually runs on".1

The reported payoff, all on the paper's own runs. On the RTX 5090 server across four agentic workloads, FreeToken sustains 77-83 tok/s on Qwen3.6-35B-A3B in BF16 (6 CPU threads) and 22-25 tok/s on DeepSeek-V4-Flash in its native MXFP4 (8 CPU threads), which the paper reports as 1.8-2.3x and 1.5-1.9x the strongest baseline in each workload, against llama.cpp, Ollama, KTransformers and MoE-Infinity serving bit-identical weight formats.1 Stability under agentic load is the sharper result: FreeToken's decode rate "stays within 12% of the single-turn W1 value across the three agent workloads", while KTransformers on DSV4-Flash "has already lost 31% of its W1 rate at W2".1 Tail TTFT separates the engines further: FreeToken's worst turn stays below 44 s in every cell, while llama.cpp reaches 232 s, Ollama 179 s, and KTransformers 946 s somewhere in the matrix, past the 120 s idle watchdog OpenClaw ships.1

Across the five consumer systems on the coding-agent workload, the paper reports leads of 1.3x on the 3090 and 4090, 1.9x on the 5090 server, 2.1x on the 5090 desktop, and 1.8x on the 4060 laptop, where an NVFP4 build sustains 39.3 tok/s on 8 GB over a x8 link. At the frontier tier, GLM-5.2 (753B-A40B, NVFP4, a 433 GB checkpoint) runs on one RTX PRO 6000 at 14.9 tok/s against llama.cpp's 7.3, with mean TTFT 7.5 s versus 7.8 s.1

When to use it (and when not)

Use it when all of these hold:

  • The expert pool exceeds VRAM but the active path does not. This is the design point. If the whole model fits, --moe-backend fused keeps experts GPU-resident and none of this machinery applies; if the active path does not fit either, no expert cache saves the deployment.
  • Single machine, single GPU, small batch. The engine's --max-running-requests defaults to 4 and the workloads evaluated are single-user agent sessions. Multi-GPU expert sharding is a different problem with a different answer (expert parallelism for MoE inference).
  • The workload is agentic. Multi-turn tool-calling traffic re-enters prefill on every turn, which is what the semantic anchors and the shared prefill/decode slot pool are built for. The paper's own W1 (single-turn AIME math, short prompts) is the one cell where llama.cpp wins on mean TTFT.1
  • B_H comfortably exceeds B_P on the target box. This is the load-bearing precondition and it is not universal. The shipped auto-selector only upgrades a machine from the plain offload backend to hybrid when the benchmarked CPU MoE bandwidth exceeds twice the PCIe gather bandwidth (recommend() in python/freetoken/moe/benchbw.py, threshold=2.0).3 Applied to the paper's own Table 1, both RTX 5090 rows fail that gate, as the executed block below asserts, but only one of them is an unambiguous failure: the 5090 desktop.4

Do not reach for it when:

  • The deployment is a datacenter GPU with the model resident. Everything here trades bandwidth for capacity; with capacity to spare it is pure overhead. Use vLLM on consumer GPUs or a standard open-weight serving path.
  • Reproducibility across machines matters. The split is a function of two bandwidths measured on that specific host, so two machines running the same checkpoint will divide the work differently. Output values still match, since the merge is exact and no experts are skipped or approximated, but timing and cache state will not.
  • The host cannot pin the expert pool. The paper's SS4.2 documents a fallback for platforms where the pool "cannot be pinned or registered for DMA (which is a restriction on some operating systems and driver configurations)": weights stay pageable and every routed expert executes on the CPU. That path "trades peak transfer bandwidth for deployability", and q* no longer applies. The CLI exposes --moe-cpu-layers for the partial version of this, explicitly citing WSL's quota-capped CUDA pinning.13
  • The model is not on the supported list. docs/models.md at the pinned commit names ten architecture families. The paper's "more than 20 MoE models" is a broader count than the repository's own known-good table, which also includes dense entries.31

Architecture

Prefill and decode use opposite strategies against the same slot pool, because prefill destroys the sparsity decode depends on: "the union of routes across a long prompt often covers most experts in every layer, turning the expert working set effectively dense".1 Prefill therefore never fetches on demand; it streams whole layers ahead of the compute. Decode fetches a computed fraction of what actually missed.

flowchart TB
    subgraph host["Host: CPU + DRAM (source of truth)"]
        POOL["complete routed-expert pool, pinned"]
        CPUX["persistent C++ worker pool, SIMD + in-kernel dequant"]
    end
    subgraph gpu["GPU: VRAM"]
        NONEXP["non-expert weights, resident"]
        SLOTS["one elastic slot pool, shared by all MoE layers"]
        KV["paged KV cache + radix prefix tree"]
    end
    POOL -->|"prefill: full layer l+1 while GPU computes layer l"| SLOTS
    SLOTS -->|"decode: LRU hits H execute on GPU"| MERGE["exact merge to layer output"]
    ROUTER["router picks top-k for token t"] --> CLASSIFY{"device kernel: dedup, classify vs residency table"}
    CLASSIFY -->|"hit"| SLOTS
    CLASSIFY -->|"m misses, split by q*"| SPLIT{"q = round(m * B_P / B_H)"}
    SPLIT -->|"F: q experts, fill over PCIe at B_P"| SLOTS
    SPLIT -->|"C: m-q experts, in place at B_H - B_P"| CPUX
    CPUX -->|"gate-weighted partial outputs"| MERGE
    BENCH["ft bench bw: measured B_P, B_H on this machine"] --> SPLIT

The split, and what it is actually optimising

Let S be the size in bytes of one complete expert. Because "both expert DMA transfers and CPU execution read from the same host-memory subsystem", a saturated PCIe transfer leaves residual host bandwidth B_R = max(B_H - B_P, 0), and the paper's Eq. 3 charges the two branches as T_fill(q) = qS / B_P and T_cpu(m-q) = (m-q)S / (B_H - B_P). Balancing them gives Eq. 4:1

q / (m - q) = B_P / (B_H - B_P)      =>      q* = m * B_P / B_H

The single ratio covers every hardware balance. As B_H approaches B_P the residual vanishes, q* approaches m, and "the system degenerates into pure on-demand cache fill without requiring separate execution branches or policies".1 The paper adds two implementation constraints: q* is rounded to an integer, the choice of which experts fill is delegated to the cache replacement policy, and the system "always retains at least one fill, so the cache continues warming even when the CPU handles most misses".1 The shipped rule has no such clamp, and this page does not resolve the gap.5

The rounding is not round(), and it is not floored at one either. The shipped kernel computes lo = floor(m * phi) in Q16 fixed point and then picks whichever of lo and lo+1 minimises max(f * (1 - phi), (m - f) * phi), which is the makespan of the two concurrent branches rescaled to be unit-free. Nothing downstream raises a zero: at the 4060 laptop's phi = 0.248 the rule returns q = 0 for m = 1, 2, 3, and the executed block below asserts the B_P = 0 endpoint as balanced_fetch(9, 0) == 0. The repository's own test names the regression this fixed: at phi = 0.415 and m = 3 the continuous optimum is 1.24, and a ceil "makes the PCIe side ~1.6x slower than balance".3 The same rule runs identically in the Triton kernel and in a CPU reference mirror, and the repo asserts they agree bit-for-bit including LRU state.

Modelling that decision in numpy exposes something the balanced form obscures. Eq. 3 charges the CPU the residual rate B_H - B_P for the whole step, even after the q fills have landed and the DMA is no longer competing. Accounting for the phases separately gives the same q*, but shows that every q from 0 to q* costs the step exactly one pass of the misses through host DRAM: total host bytes are mS whichever way the work is split, so mS / B_H is a floor no split can beat. On that reading q* is not a latency optimum at all, it is the largest number of cache fills that is free on the current step, and its value is entirely in the future hits the residency buys. The repository independently rejects the same full-contention assumption: its preferred fetch fraction comes from measuring both sides while they contend, because assuming full contention "over-penalizes a CPU kernel that never saturated DRAM to begin with".3

"""FreeToken's decode-time miss split (q*), modelled and asserted in numpy.

Reimplements the integer split that FreeToken's device-side kernel computes per MoE
layer per decode step (FlashML-org/FreeToken @ 9ef3651, python/freetoken/moe/
offload_kernels.py, _ensure_experts_hybrid_cpu and _ensure_experts_lru_hybrid_kernel),
then checks it against the paper's Eq. 4 (arXiv 2608.16157 SS3.2), against a
brute-force optimum, at both bandwidth endpoints, on the paper's Table 1 machines,
and on two cases where the design's premises stop holding.
"""

import numpy as np

Q16 = 1 << 16

# ---------------------------------------------------------------- 1. the split rule

def balanced_fetch(m: int, frac_q16: int) -> int:
    """FreeToken's per-(layer, step) fetch count q, in the repo's exact Q16 integers.

    cost(f) = max(f*(1-phi), (m-f)*phi) is the makespan of the two concurrent branches
    of paper Eq. 3, rescaled by phi*(1-phi)*B_H/S so it is unit-free."""
    lo = (m * frac_q16) >> 16
    cost = lambda f: max(f * (Q16 - frac_q16), (m - f) * frac_q16)
    pick = lo if cost(lo) <= cost(lo + 1) else lo + 1
    return min(m, pick)


def brute_force_fetch(m: int, phi: float) -> int:
    """Argmin over every legal integer q of the same makespan."""
    costs = [max(f * (1.0 - phi), (m - f) * phi) for f in range(m + 1)]
    return int(np.argmin(costs))


for phi in (0.1, 0.2484, 0.3972, 0.415, 0.6817, 0.9108, 1.0):
    fq = round(phi * Q16)
    for m in range(0, 65):
        q = balanced_fetch(m, fq)
        assert 0 <= q <= m
        # paper Eq. 4 is q* = m * B_P/B_H; the integer rule never drifts past one expert
        assert abs(q - phi * m) <= 1.0, (phi, m, q)
        # and it is the true integer optimum of the makespan, not just a rounding
        assert np.isclose(max(q * (1 - phi), (m - q) * phi),
                          max(brute_force_fetch(m, phi) * (1 - phi),
                              (m - brute_force_fetch(m, phi)) * phi)), (phi, m)
# the repo's own regression: naive ceil over-fetches at m=3, phi=0.415 (1.24 -> 2)
assert balanced_fetch(3, round(0.415 * Q16)) == 1
assert balanced_fetch(4, round(0.415 * Q16)) == 2
assert int(np.ceil(0.415 * 3)) == 2  # what the rule deliberately does not do
print("[1] split rule: matches Eq.4 within 1 expert, is the integer argmin, ceil-trap avoided")

# --------------------------------------------- 2. what the split actually costs a step

def makespan_paper(m: int, q: int, S: float, bp: float, bh: float) -> float:
    """Paper Eq. 3 verbatim: the CPU is charged the residual rate for the whole step."""
    t_fill = q * S / bp
    t_cpu = np.inf if bh <= bp else (m - q) * S / (bh - bp)
    return max(t_fill, t_cpu if m > q else 0.0)


def makespan_phased(m: int, q: int, S: float, bp: float, bh: float) -> float:
    """Phase-aware accounting: the CPU is throttled to B_H - B_P only while the DMA is
    actually in flight, and regains all of B_H once the q fills have landed."""
    t_fill = q * S / bp if q else 0.0
    work = (m - q) * S
    done = max(bh - bp, 0.0) * t_fill
    if done >= work:
        return t_fill
    return t_fill + (work - done) / bh


S, bp, bh, m = 34.0e6, 52.7e9, 77.3e9, 12          # RTX 5090 row of Table 1
phi = bp / bh
qs = np.arange(m + 1)
paper = np.array([makespan_paper(m, q, S, bp, bh) for q in qs])
phased = np.array([makespan_phased(m, q, S, bp, bh) for q in qs])
q_star = balanced_fetch(m, round(phi * Q16))

assert q_star == int(np.argmin(paper)) == 8, (q_star, np.argmin(paper))
# Under the phase-aware model every q from 0 to q* costs exactly one pass of the misses
# through host DRAM: the host bus, not the split, is the floor.
floor = m * S / bh
assert np.allclose(phased[: q_star + 1], floor, rtol=1e-12)
assert np.all(np.diff(phased[q_star:]) > 0)            # strictly worse past q*
assert phased[m] / floor > 1.46                        # pure PCIe fill: B_H/B_P slower
print(f"[2] 5090 (B_P {bp/1e9:.1f}, B_H {bh/1e9:.1f} GB/s), m={m}: q*={q_star}; "
      f"phase-aware step cost flat at {floor*1e3:.3f} ms for q<=q*, "
      f"{phased[m]*1e3:.3f} ms at q=m ({phased[m]/floor:.2f}x)")
print("    -> q* is the largest number of cache fills that is free on this step")

# ---------------------------------------------------------------- 3. both endpoints

# B_P >= B_H: no residual host bandwidth exists, the policy must degenerate to pure fill
for bp_e, bh_e in ((80e9, 77.3e9), (77.3e9, 77.3e9), (200e9, 47.5e9)):
    fq = min(Q16, round(bp_e / bh_e * Q16))
    assert balanced_fetch(9, fq) == 9
    assert makespan_paper(9, 9, S, bp_e, bh_e) < makespan_paper(9, 8, S, bp_e, bh_e)
# B_P = 0 (no usable DMA path: the paper's SS4.2 unpinnable-host fallback): pure CPU
assert balanced_fetch(9, 0) == 0
assert np.isclose(makespan_phased(9, 0, S, 1.0, bh), 9 * S / bh)
print("[3] endpoints: B_P>=B_H -> q*=m (pure offload); B_P=0 -> q*=0 (pure CPU MoE)")

# ------------------------------------------- 4. Table 1 machines vs the shipped gate

# System, B_P, B_H (GB/s), paper Table 1. recommend() in python/freetoken/moe/benchbw.py
# only upgrades a machine to the hybrid backend when B_H > threshold * B_P, default 2.0.
TABLE1 = [("5090 server", 52.7, 77.3), ("4090", 25.1, 63.2), ("3090", 25.3, 56.7),
          ("5090 desktop", 49.0, 53.8), ("4060 laptop", 11.8, 47.5),
          ("PRO 6000", 51.5, 178.0)]
gated, rows = {}, []
for name, p, h in TABLE1:
    f = p / h
    gated[name] = h > 2.0 * p
    rows.append((name, f, balanced_fetch(4, round(f * Q16)), h / p, gated[name]))
    print(f"    {name:<13} phi={f:.3f}  q*(m=4)={rows[-1][2]}  B_H/B_P={h/p:.2f}  "
          f"hybrid gate: {'pass' if gated[name] else 'FAIL -> pure offload'}")
# The two machines carrying the headline decode numbers are exactly the two the shipped
# default gate rejects, so they run the offload backend and never reach the q* path.
assert not gated["5090 server"] and not gated["5090 desktop"]
assert all(gated[n] for n in ("4090", "3090", "4060 laptop", "PRO 6000"))
assert rows[0][2] == 3 and rows[4][2] == 1     # 5090 would fetch 3 of 4; laptop 1 of 4
print("[4] Table 1 vs benchbw.recommend(threshold=2.0): both 5090 rows fail the gate")

# ----------------------------------- 5. adversarial: where batching the misses buys 0

rng = np.random.default_rng(0)


def unique_routed(batch: int, k: int, E: int, weights: np.ndarray, trials: int = 20000) -> float:
    """Mean distinct experts one MoE layer touches for a batch of `batch` tokens."""
    out = np.empty(trials)
    for t in range(trials):
        picks = [rng.choice(E, size=k, replace=False, p=weights) for _ in range(batch)]
        out[t] = len(set(np.concatenate(picks).tolist()))
    return out.mean()


E, k = 256, 6                                   # DeepSeek-V4-Flash: 6 of 256 routed
uniform = np.full(E, 1.0 / E)
zipf = 1.0 / (np.arange(1, E + 1) ** 1.2)
zipf /= zipf.sum()

# batch=1 is FreeToken's edge case and there is nothing to deduplicate: gain is exactly 1
assert unique_routed(1, k, E, uniform, 400) == float(k)
assert unique_routed(1, k, E, zipf, 400) == float(k)
# at the engine's default --max-running-requests 4, uniform routing barely collides, so
# per-step batch dedup is worth ~3%; even heavy skew returns under 1.4x. The miss
# reduction is not coming from here, it is the cross-step LRU (paper SS5.3: 16% / 39%)
u4 = unique_routed(4, k, E, uniform, 4000)
z4 = unique_routed(4, k, E, zipf, 4000)
gain_u, gain_z = 4 * k / u4, 4 * k / z4
# closed form for the uniform case: E[unique] = E * (1 - (1 - k/E)^B)
assert np.isclose(u4, E * (1 - (1 - k / E) ** 4), rtol=2e-3), (u4,)
assert 1.02 < gain_u < 1.05, gain_u
assert gain_z > gain_u                          # skew helps, uniform routing does not
assert 1.30 < gain_z < 1.50, gain_z
print(f"[5] batch dedup at B=4, E=256, k=6: uniform {gain_u:.3f}x, Zipf(1.2) {gain_z:.3f}x; "
      f"at B=1 it is exactly 1.000x")
print("    -> on single-stream edge decode the split has no batching to hide behind")

# ------------------------------- 6. adversarial: q* on a machine with no CPU MoE path
# If the CPU kernel underperforms its DRAM (SS4.2 falls back to pageable host memory),
# the measured B_H collapses below B_P and Eq. 4 saturates: the formula cannot express
# "fetch less than everything" once the host cannot beat the link.
assert balanced_fetch(6, min(Q16, round((52.7 / 20.0) * Q16))) == 6
assert makespan_phased(6, 6, S, 52.7e9, 20.0e9) < makespan_phased(6, 0, S, 52.7e9, 20.0e9)
print("[6] degraded CPU path (B_H 20 < B_P 52.7 GB/s): q*=m, hybrid collapses to offload")
print("all assertions passed")

Executed output (python3, numpy 2.4.6; expert size 34 MB and m are this page's illustrative inputs, the bandwidths are the paper's Table 1):

[1] split rule: matches Eq.4 within 1 expert, is the integer argmin, ceil-trap avoided
[2] 5090 (B_P 52.7, B_H 77.3 GB/s), m=12: q*=8; phase-aware step cost flat at 5.278 ms for q<=q*, 7.742 ms at q=m (1.47x)
    -> q* is the largest number of cache fills that is free on this step
[3] endpoints: B_P>=B_H -> q*=m (pure offload); B_P=0 -> q*=0 (pure CPU MoE)
    5090 server   phi=0.682  q*(m=4)=3  B_H/B_P=1.47  hybrid gate: FAIL -> pure offload
    4090          phi=0.397  q*(m=4)=1  B_H/B_P=2.52  hybrid gate: pass
    3090          phi=0.446  q*(m=4)=2  B_H/B_P=2.24  hybrid gate: pass
    5090 desktop  phi=0.911  q*(m=4)=4  B_H/B_P=1.10  hybrid gate: FAIL -> pure offload
    4060 laptop   phi=0.248  q*(m=4)=1  B_H/B_P=4.03  hybrid gate: pass
    PRO 6000      phi=0.289  q*(m=4)=1  B_H/B_P=3.46  hybrid gate: pass
[4] Table 1 vs benchbw.recommend(threshold=2.0): both 5090 rows fail the gate
[5] batch dedup at B=4, E=256, k=6: uniform 1.035x, Zipf(1.2) 1.377x; at B=1 it is exactly 1.000x
    -> on single-stream edge decode the split has no batching to hide behind
[6] degraded CPU path (B_H 20 < B_P 52.7 GB/s): q*=m, hybrid collapses to offload
all assertions passed

Three results carry beyond the formula. Block 4 is the uncomfortable one: applying the repository's own default hybrid gate to the paper's own Table 1, the two RTX 5090 rows that carry the headline decode numbers fail it, so a user reproducing on those Table 1 numbers gets the offload backend and never exercises q* at all. The arithmetic is exact but the reading needs one qualification: the 5090 server is a rented dual-socket box that the paper deliberately throttled to 6 CPU threads, so its 77.3 GB/s is an emulated edge figure, and the same box benchmarked with ft bench bw's defaults (one thread per physical core) could well clear the gate. The 5090 desktop is the clean case: 53.8 GB/s at its natural 16 cores against 49.0 GB/s of PCIe, a ratio of 1.10 that no threading choice rescues.4 Block 5 kills a tempting explanation for the miss-rate win: at the engine's default four concurrent requests over 256 experts with top-6 routing, deduplicating misses across the batch returns 1.035x under uniform routing and 1.377x under heavy skew, and at batch 1 it returns exactly nothing. The reported miss reduction has to come from the cross-step LRU, not from batching. Block 6 is the degenerate case: once a degraded CPU path drops B_H below B_P, Eq. 4 saturates at q = m and hybrid silently becomes plain offload, with no error and no log line about it.

Prefill: full-layer double buffering

Because prefill activates nearly the whole expert set, FreeToken "does not fetch prefill experts on demand". It takes two full-layer buffers from the same global slot pool; while the GPU computes layer l from one, a dedicated transfer stream loads the complete expert set of layer l+1 into the other, which means the transfer can start "before that layer's routing is even known".1 Sharing the decode cache's slot pool means there is no phase handoff, and entries surviving prefill seed the decode cache.

The measured effect (RTX 5090, Qwen3.6-35B BF16): with overlap on, each 8192-token prefill chunk completes in 1.19-1.22 s, which the paper identifies as the time to stream the 64.4 GB expert pool once at 52.7 GB/s, so expert computation is fully hidden and prefill throughput climbs to 6.7k tok/s at 16k tokens. Disabling the second buffer costs 19% at 4k tokens, 25% at 8k, and 26% at 16k.1 The guard is memory: the pool must spare two full layers, and the shipped code disables overlap whenever cache_size < 2 * num_experts (asserted at construction, warned and disabled on rebuild).3

Decode residency: one LRU across all layers

Replayed on identical routing traces at equal cache capacity (37% of Qwen3.6's expert pool, 11% of DSV4-Flash's, the RTX 5090 serving capacity), FreeToken's global LRU misses 16% and 39% of decode-time expert reads, against 41% and 59% for KTransformers' prefill-updated placement and 62% and 89% for llama.cpp's routing-blind static split.1 The mechanism it exploits is short-range routing consistency, measured independently across model families.[^routing-consistency]

All routing-dependent control stays on the GPU so the whole step remains inside a statically captured CUDA Graph: one kernel per MoE layer deduplicates the routed experts, classifies them against the residency table, derives q, selects eviction victims and rewrites logical expert IDs into physical slot IDs or a CPU-assignment flag. Victim selection "avoids the classic LRU trap of requiring one full-cache scan per evicted slot": a single pass finds the K least-recently-used candidates, and the miss path consumes the first q <= K, so "victim discovery always costs exactly one pass, regardless of the realized miss count".1 In the shipped code the overflow misses are marked by rewriting their entry to -1.3 The CPU branch is captured into the same graph: pinned I/O buffers and persistent task descriptors per supported decode batch size, a host-function submit node, a synchronisation node, and the result copy, all captured together.

Semantic anchors for recurrent state

Hybrid-attention models add a prefix resource the KV cache does not cover: a gated-DeltaNet or Kimi-Delta layer "compresses its entire prefix into one evolving state that cannot be partially reused", so reuse depends on state checkpoints, and each checkpoint costs as much memory as the KV of hundreds of tokens. FreeToken spends that budget at semantic anchors, the special-token boundaries agent harnesses actually edit at, citing three concrete behaviours: OpenClaw "strips thinking blocks from every assistant turn but the latest", OpenCode "replaces tool outputs beyond a recent window with a fixed placeholder", and SWE-agent "elides all but the last n observations". Checkpoint slots are recycled with LRU eviction, independently of the KV pool.1

The shipped implementation is narrower than that description, and off unless --enable-special-token-ckpt is passed. This is the largest paper-versus-code gap found.7

How to use it

Reference templates against commit 9ef3651, unexecuted here. Version pins from pyproject.toml at that commit: Python >=3.10, torch>=2.11,<2.12, numpy>=2.0,<2.5; the sole tagged release is v0.1.2 (2026-08-19).

# install (uv recommended by the README)
uv pip install "freetoken[accel]"

# serve; --model is the only required flag, everything else auto-resolves
ft serve --model Qwen/Qwen3.6-35B-A3B          # OpenAI + Anthropic APIs on 127.0.0.1:1919

# point a coding agent at it
ft launch claude                               # claude / codex / dsh / hermes / openclaw / opencode

The q* path is not on by default. --moe-backend auto resolves MoE models to offload, and upgrades to hybrid only when a cached bandwidth profile recommends it. Producing that profile is a manual, one-time, per-machine step:

ft bench bw                                    # measures B_P and B_H, writes a per-GPU profile
ft serve --model <path> --moe-backend hybrid   # or leave it on auto once the profile exists

The knobs that matter, from docs/cli.md and python/freetoken/server/args.py at the pinned commit:3

Flag Default What it controls
--moe-backend auto fused / offload / cpu / hybrid. auto picks offload, upgraded to hybrid only with a ft bench bw profile that passes the 2x threshold
--moe-hybrid-max-fetch -1 (auto) Max experts fetched over PCIe per layer per step. -1 uses the benched fraction (q*); an explicit integer pins a fixed cap; 0 never fetches
--moe-cache-size / --moe-cache-rate / --moe-cache-auto auto Expert cache as slots, as a fraction of all experts, or sized from free VRAM. Mutually exclusive
--kv-reserve-tokens 8192 KV token floor reserved before --moe-cache-auto fills the rest with experts
--moe-cpu-threads physical cores Worker threads for the cpu / hybrid executor
--moe-cpu-layers None (automatic) Which MoE layers decode on the CPU executor instead of the PCIe path. Unset means automatic where CUDA pinning is quota-capped (WSL): head and tail layers are locked only when the banks exceed the pin budget, none otherwise. 0 is what forces all layers on GPU. List, count, or fraction6
--disable-moe-prefill-overlap overlap on Turns off the two-buffer prefill pipeline
--enable-special-token-ckpt off Opt-in. Checkpoints decode state at the tool-call opener, on GDN-hybrid or SWA models only. The semantic anchor does nothing without it7
--memory-ratio 0.9 Fraction of free VRAM the engine may claim in total
--max-running-requests 4 Concurrent requests, which also bounds the captured CUDA-graph batch sizes

Read --moe-hybrid-max-fetch carefully: -1 is the paper's policy, any non-negative integer is a fixed cap, and with no usable profile the engine logs a warning and falls back to a fixed cap of one fetch per layer per step, which is not q*.3

How to develop with it

The repository is Apache-2.0 (LICENSE, 201 lines by wc -l, 169 of them non-blank; standard Apache 2.0 text). Source layout: python/freetoken/moe/ holds everything discussed here (offload_cache.py for the slot cache and its dataclass knobs, offload_kernels.py for the Triton residency kernel and its CPU reference mirror, cpu_executor.py for the worker pool, benchbw.py and bench_profile.py for the bandwidth profile), engine/ for config and the CUDA-graph capture, scheduler/ for the anchor logic, server/ for the API surfaces.

git clone https://github.com/FlashML-org/FreeToken.git && cd FreeToken
git checkout 9ef3651309fe4058672f2cc92069238dea06be1b     # the commit this page reads
uv venv && source .venv/bin/activate
uv pip install -e ".[accel]"
pytest tests/moe/test_hybrid_fetch.py                     # the split's own tests

tests/moe/test_hybrid_fetch.py is the file to read before touching the policy. It covers both halves: the profile reader that turns benchmark bandwidths into a fetch fraction, and the per-step integer split, with a CUDA-gated test asserting the Triton kernel and the CPU reference agree on the fetch count, the slot rewrites, and the full LRU state over 64 steps. Most of the CUDA-dependent tests skip without a GPU.

Two facts about the split are worth knowing before changing it. The fetch fraction is computed once at engine start, not per step, and is stored on the cache object; the per-step work is only the integer rounding, done on device. And the fraction has two derivations in load_hybrid_fetch_fraction: the preferred one uses a contended measurement pair, pcie_ov / (pcie_ov + cpu_ov), and only profiles lacking it fall back to pcie / cpu, which is Eq. 4 exactly.3 These coincide when contention is total; they diverge when the CPU kernel never saturated DRAM.

How to maintain it

  • The bandwidth profile is machine state, and it goes stale. It is written per GPU (keyed by UUID, with a legacy name-keyed file as fallback) and read at every engine start. A CPU upgrade, a RAM channel change, a BIOS PCIe-link change, a kernel rebuild that changes SIMD dispatch, or a container CPU-quota change all invalidate it silently. Re-run ft bench bw after any of them.
  • Confirm the backend actually chosen. The upgrade to hybrid is unanimous-or-nothing: load_backend_recommendation returns hybrid only when every benched workload sharing that expert format recommended it, and a mixed verdict resolves to offload.3 Check the startup log for the --moe-hybrid-max-fetch auto: fetching N% of each decode step's expert misses over PCIe line; its absence, or the paired "no usable profile" warning, means the deployment is on a fixed cap of 1.
  • The profile is per expert format, not per model. Formats map through _QUANT_TO_BENCH_FORMAT; a machine benched for bf16 has nothing to say about an nvfp4 checkpoint, and the reader returns None rather than guessing.
  • Track upstream closely. At the pin the repository is about five weeks old (created 2026-07-20), carries 8,065 stars, 699 forks and 178 open issues, and had commits landing the same day the pin was taken. One tagged release exists (v0.1.2, five wheel assets). Treat flag names and defaults as unstable; the commit two before the pin is a breaking change adding --gpu.3

How to run it in production

"Production" here means a workstation or a single-tenant box, not a fleet. The engine assumes it shares the machine.

  • Resize instead of restarting. The elastic cache is exposed over HTTP (POST /v1/cache/rebuild) and the CLI (ft ctl cache rebuild, plus /cache inside ft shell). A rebuild pauses serving, runs only when idle, clears the prefix cache, and resets hit/miss counters, so post-rebuild statistics start cold by design. Use it when another application claims VRAM, or when a long agent session has grown KV demand past the split chosen at launch.3
  • Budget KV explicitly on agent workloads. --moe-cache-auto is MoE-priority: it fills experts and gives KV only --kv-reserve-tokens as a floor. The paper's W3 workload grows sessions to 56-65k tokens and W4 carries a roughly 24.5k-token system-context floor, so the 8192-token default floor is far below what an agent session actually needs. Raise it or size KV explicitly with --num-tokens.
  • Watch tail TTFT, not mean TTFT. The paper's own framing is that "tail TTFT is therefore an availability boundary, not a latency statistic", with OpenClaw's 120 s idle watchdog and Claude Code's roughly ten-minute default request timeout as the thresholds that turn a slow turn into a failed one.1 Alert on the p99, not the mean.
  • Startup is a user-visible cost and recurs. For an FP4 DeepSeek-V4-Flash deployment, reading the roughly 140 GB pool from a 7 GB/s NVMe drive alone takes about 20 seconds before any warmup.1 FreeToken loads directly into the final host layout and pins only afterwards ("pinning empty buffers first would fault in and zero gigabytes of pages merely to overwrite them"), and skips GPU warmup entirely by serving the first request cold. ft checkpoint pre-converts to the FTW bank layout, which skips tensor discovery and repacking; do it once, offline, not on the serving path.
  • No FTW checkpoints are published. ft checkpoint converts a local HF checkpoint; the conversion is the operator's to run and to store. Plan the disk for both copies.
  • Expect a GPU-only deployment surface. benchbw refuses to run without CUDA ("benchbw needs a CUDA device to measure PCIe bandwidth"), and the README claims native support for RTX 30, 40 and 50 series. There is no AMD or Apple path here; for those see AMD ROCm inference serving.

Failure modes

Symptom Cause Action
hybrid never engages; decode is PCIe-bound auto resolved to offload: no ft bench bw profile, a profile from a different GPU, or B_H <= 2 * B_P on this box Run ft bench bw on the box itself rather than reading Table 1: the 5090 server row there is a 6-thread emulation, while the 5090 desktop's 1.10 ratio fails at full threads. If the measured ratio genuinely fails the 2x gate, offload is the correct answer, not a bug
Startup warns "no usable ft bench bw profile"; throughput below expectation Engine fell back to a fixed cap of one PCIe fetch per layer per step, not q* Generate the profile for the exact expert format being served; the reader is format-keyed and returns nothing for an unbenched format
Prefill TTFT regresses by roughly 20-26% Prefill overlap disabled because cache_size < 2 * num_experts. At construction this is a hard assert that raises; the silent paths are --moe-cache-auto budget resolution (cache_budget.py: overlap = prefill_overlap and hi >= 2 * num_experts) and split-residency setup, with a warn-and-disable on rebuild Grep the log for "Disabling MoE prefill overlap"; raise --moe-cache-size or lower --kv-reserve-tokens
Hybrid behaves exactly like offload, no error B_H measured below B_P (degraded CPU kernel, pageable fallback, thread-capped container), so Eq. 4 saturates at q = m Compare the profile's cpu_moe_gbs against pcie_gather_gbs; check --moe-cpu-threads and NUMA pinning
Every expert executes on the CPU; PCIe idle Host pool could not be pinned or DMA-registered (OS or driver restriction), triggering the pure-CPU MoE backend Check pinning limits; on WSL use --moe-cpu-layers to route only the quota-affected layers
Two identical machines diverge in throughput Each measured its own B_P and B_H, so each computed a different split; a stale profile on one of them widens the gap Re-run ft bench bw on both and diff the profiles. Outputs stay identical (exact merge, no expert skipping); only timing differs
Cache statistics reset mid-session A /v1/cache/rebuild deliberately zeroes hit/miss counters and clears the prefix cache Expected. Re-baseline monitoring after any rebuild
Agent session slows as it grows KV demand grew past the launch-time split while the expert working set stayed fixed Rebuild the cache at the new budget, or raise --kv-reserve-tokens up front
No model served on a listed architecture Multimodal checkpoints are served text-only; DeepSeek-V4 needs its inference/config.json subdir preserved Check docs/models.md for the per-family caveats

References

Related: MoE routing and expert load balancing, MoE expert backends and grouped-GEMM kernels, Mixture-of-experts: sparse scaling, expert parallelism for MoE inference, Colibri: GLM-5.2 on a 25GB-RAM consumer box, DwarfStar (ds4): DeepSeek V4 local inference, DeepSeek-V4-Flash on your own hardware, vLLM: small models on consumer GPUs, local coding agents, serving open-weight models, quantization for inference, KV cache and inference speedup

  1. arXiv:2608.16157v1. Architecture and the two-level hierarchy, SS3 opening; prefill costs and the 140 GB / 2 s / 5 s / 10 s transfer figures, SS2.1; the static-placement and CPU-bandwidth arguments, SS2.2; q*, Eqs. 1-4 and the "always retains at least one fill" rule, SS3.2; semantic anchors and the three harness behaviours, SS3.1; the pure-CPU pinning fallback, SS4.2; hardware Table 1 and its caption; decode throughput, the 12% stability figure and the KTransformers 31% loss, SS5.2; tail TTFT figures (44 s / 232 s / 179 s / 946 s), SS5.2; prefill overlap and miss-rate breakdowns (1.19-1.22 s, 6.7k tok/s, 19% / 25% / 26%, 16% / 39% vs 41% / 59% vs 62% / 89%) and the cross-hardware multipliers, SS5.3. Note two different expert-pool sizes appear for different models: roughly 140 GB for FP4 DeepSeek-V4-Flash (SS2.1, SS2.3) and 64.4 GB for Qwen3.6-35B BF16 (SS5.3). ↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩↩

  2. README.md at commit 9ef3651. Its headline reads "Run 290B+ frontier MoE models locally on your gaming PC", while the paper consistently describes the same demonstration as a 284B model (DeepSeek-V4-Flash, 284B total / 13B active) on a 32 GB gaming desktop. The two figures are not reconciled by either source. ↩

  3. FlashML-org/FreeToken at commit 9ef3651309fe4058672f2cc92069238dea06be1b. The 2x hybrid gate and the "unanimous or offload" aggregation: python/freetoken/moe/benchbw.py (recommend, default threshold=2.0) and python/freetoken/moe/bench_profile.py (load_backend_recommendation). The two fetch-fraction derivations: bench_profile.py::load_hybrid_fetch_fraction, preferring pcie_ov / (pcie_ov + cpu_ov) from measure_overlap_bw and falling back to pcie / cpu; the docstring of measure_overlap_bw states that the full-contention assumption "over-penalizes a CPU kernel that never saturated DRAM to begin with", which is precisely the assumption the paper's Eq. 2 encodes. The fixed-cap-of-1 fallback and its warning: python/freetoken/engine/engine.py::_resolve_hybrid_fetch. The integer split and the -1 overflow marker: python/freetoken/moe/offload_kernels.py (_ensure_experts_hybrid_cpu, _ensure_experts_lru_hybrid_kernel); the ceil regression it fixes is named in tests/moe/test_hybrid_fetch.py::test_balanced_fetch_tracks_fraction. Prefill-overlap guard: python/freetoken/moe/offload_cache.py (assert at construction, warn-and-disable in rebuild). Flags: python/freetoken/server/args.py and docs/cli.md. Supported models: docs/models.md (ten table rows: DeepSeek-V4, GLM-5.2, GLM-4.7, Qwen3.6 / Qwen3.5 MoE, Qwen3.6 dense, Qwen3-MoE, gpt-oss, Gemma-4, MiniMax-M2.5, Muse-Glimmer; two of them dense). Repository statistics from the GitHub API on 2026-08-26. ↩↩↩↩↩↩↩↩↩↩↩↩↩

  4. Paper SS5.1, "Hardware": "The 3090, 4090, and 5090 systems are rented dual-socket servers whose CPUs far exceed any edge host, so every serving run and bandwidth measurement on them is capped at 6 CPU threads and pinned to the GPU's NUMA node. Capped this way, the servers deliver 56.7-77.3 GB/s of host bandwidth, the same scale the two real edge machines reach at their natural full threads (53.8 GB/s on the desktop's 16 cores, 47.5 GB/s on the laptop's 14)." So three of the six Table 1 rows are throttled emulations. For the 4090 and 3090 that only strengthens their pass (an unthrottled B_H would be higher still), but for the 5090 server it means the 77.3 GB/s that fails the 2x gate is a floor imposed by the paper, not a property of the box. python/freetoken/moe/benchbw.py::measure_cpu_moe_bw takes num_threads: int = 0, and the CLI documents 0 as "one per physical core" via resolve_threads_and_affinity, so a default ft bench bw run on that server measures at full physical-core count. Two further mismatches between Table 1 and the gate: recommend() is evaluated per (workload, expert format) inside _bench_format against the synthetic canonical geometries in DTYPE_WORKLOADS / WORKLOADS, and load_backend_recommendation requires every benched workload sharing a format to agree, whereas Table 1 publishes one bandwidth pair per machine "measured on the deployed tensor shapes". The executed block's gate arithmetic is exact on Table 1's published numbers; it is not a prediction of what ft bench bw will return on any given box. ↩↩

  5. Paper-versus-code gap, recorded not resolved. Paper SS3.2: "In practice, FreeToken rounds q⋆ to an integer and delegates the specific selection of F to the cache replacement policy. It always retains at least one fill, so the cache continues warming even when the CPU handles most misses." At commit 9ef3651 neither implementation of the split enforces that floor. python/freetoken/moe/offload_kernels.py::_ensure_experts_hybrid_cpu computes lo = (m * frac_q16) >> 16, picks lo or lo + 1 by the makespan comparison, then num_fetch = min(len(missing), int(max_fetch)); _ensure_experts_lru_hybrid_kernel performs the identical arithmetic in Triton and ends at num_fetch = tl.minimum(num_missing, max_fetch). Neither takes a maximum against 1. Concretely, at the 4060 laptop's phi = 11.8 / 47.5 = 0.248 the rule returns q = 0 for every m from 1 to 3, so on a step whose misses are few the cache does not warm at all. This may be an unshipped clamp rather than a misstatement; the repository's tests do not cover the small-m case. ↩

  6. Docs-versus-code discrepancy on --moe-cpu-layers, resolved in favour of the code per this repo's rule. docs/cli.md line 86 tabulates the default as "all on GPU". The code disagrees: python/freetoken/engine/config.py declares moe_cpu_layers: str | None = None, python/freetoken/server/args.py passes that through as the argparse default, and its own help text says "Unset = automatic where CUDA pinning is quota-capped, e.g. WSL (locks just enough head+tail layers when the banks exceed the pin budget, none otherwise); '0' forces all layers on GPU." Read at commit 9ef3651. ↩

  7. Paper SS3.1 describes checkpoints anchored at "thinking segments, tool calls and outputs, and conversation turns", taken during prefill, held in a pool with independent LRU eviction. At commit 9ef3651 the shipped semantic anchor is one boundary only: the first sampled tool-call opening token. It is also off by default. python/freetoken/scheduler/scheduler.py populates self.toolcall_anchor_id only under if config.special_token_ckpt and (self.cache_manager.is_hybrid or self.cache_manager.is_swa), and python/freetoken/scheduler/config.py declares special_token_ckpt: bool = False, exposed as the opt-in --enable-special-token-ckpt in python/freetoken/server/args.py. The gate is architecture-disjunctive, not hybrid-only: hybrid recurrent models take the state-snapshot path in python/freetoken/scheduler/cache.py::snapshot_toolcall_anchor (which returns immediately on if not self.is_hybrid: return), while SWA models use the same req.toolcall_anchor_len in maybe_free_swa_out_of_window to keep the window ending at the anchor resumable, dropping the anchor if decode runs more than window + _SWA_RETAIN_GAP past it. scheduler.py sets req.toolcall_anchor_len on that token during decode (not batch.is_prefill). python/freetoken/utils/hf.py::load_toolcall_anchor_id further restricts it: the opener must tokenize to exactly one id, "so only a one-token opener can anchor", and returns None otherwise. No thinking-segment, tool-output, or conversation-turn anchor exists in the tree at this commit. General (non-semantic) recurrent-state reuse does exist through chunk-granular snapshots sized by linear_state_cache_ratio (default 2.0). This page does not resolve the gap; it may be unshipped work rather than a misstatement. ↩↩