Markdown

Attention-FFN disaggregation (AFD) for MoE serving¶

Scope: the third level of inference disaggregation, below chunked-prefill aggregation and prefill/decode splitting. AFD places the attention operators and the MoE-FFN operators of the same layer on physically separate GPU pools, then sizes those pools independently. This page covers when that pays, how to pick the attention-to-FFN ratio, what the communication pattern costs, and why the technique is still a research prototype rather than a serving flag. Phase-level splitting is in disaggregated inference and its sizing rule in rate matching; expert placement is in expert parallelism for inference.

Findings here come from Wu et al., "How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving" (arXiv 2605.28302, submitted 2026-05-27). Its cluster-scale numbers are model-based design-space-exploration estimates, produced by fusing measured kernel costs with a packet-level network simulator, not end-to-end measurements on a 128-GPU cluster. The authors' vLLM prototype was used to verify functional correctness of the execution path, not to produce the throughput figures.¹ The Python block below is this page's own scheduling model, executed and asserted; it reproduces the paper's Equation 1 and its placement arithmetic, and it is explicit about which results are the page's rather than the paper's.

flowchart TB
  subgraph AP["Attention pool (N_A GPUs)"]
    ATT["Attention + KV cache + MoE router + shared experts"]
  end
  subgraph FP["FFN pool (N_F GPUs)"]
    EXP["Routed expert FFNs (no KV cache)"]
  end
  ATT -->|"A2F dispatch: hidden states, token ids, routing metadata (fan-out)"| EXP
  EXP -->|"F2A combine: partial expert outputs (fan-in)"| ATT
  ATT -->|"next layer"| ATT
  FP -.->|"holds most model weights"| AP

What it is¶

Modern Transformer blocks are internally heterogeneous. Attention is memory-bandwidth-bound: it streams the KV cache and moves far more bytes than it does arithmetic. The MoE-FFN is compute-bound: dense GEMMs against expert weights, with high arithmetic intensity. Aggregated serving and prefill/decode disaggregation both treat a layer as one indivisible unit, so a single GPU pool has to satisfy both profiles at once, and the provisioning ratio is fixed at whatever the model architecture happens to imply.

AFD breaks that coupling. Attention executes on one set of GPUs, the routed experts on another, and the two exchange tokens twice per layer. The MoE dispatch and combine operators, which in a conventional expert-parallel deployment move tokens among peer ranks, become transfers between two differently sized pools. That asymmetry is the whole point: a deployment can now buy attention capacity and FFN capacity in whatever proportion the workload actually needs.

The concrete mechanism, as prototyped on vLLM v0.16.0rc2 by the paper's authors:²

All-pairs bipartite pairing. Every attention rank is paired with every FFN rank through its own NCCL pair-group, giving N_A * N_F communicators. Bootstrap uses a per-pair Gloo rendezvous on unique TCP ports so the communicators can be built without deadlocking or disturbing the global process group's counter state. This replaces an earlier 1:1 pairing that only supported symmetric N_A == N_F layouts.
Attention-side routing. The MoE router (gate projection plus top-k selection) and the shared experts move from the FFN side to the attention side. Each attention rank decides routing locally, ships the post-attention hidden state with token ids and per-expert routing metadata to each paired FFN partner, and sums the returned partial tensors element-wise. Consequently there is no FFN-to-FFN collective at all: the entire MoE path is point-to-point on the bipartite graph, the same shape adopted by StepFun's StepMesh transport library.³
Expert-kernel bypass. vLLM's normal FusedMoE.quant_method.apply path descends into FusedMoEModularKernel.forward, whose prepare_finalize hooks perform exactly the expert-parallel all-gather and reduce-scatter that attention-side routing exists to avoid. The prototype adds a FusedMoE.forward_pre_routed entry point that calls the raw fused_experts Triton kernel directly with expert_map filtering, yielding a zero-collective MoE layer on the FFN side.
Two transport modes. A dense mode ships the full post-attention payload on every link and lets expert_map zero out unmatched tokens. An opt-in sparse pre-routing mode pre-masks on GPU and sends only tokens whose top-k experts hit an expert local to the receiving rank. The authors state both modes produce bit-identical FFN outputs given identical routing.

Why use it¶

Independent sizing is the entire value proposition, and it shows up in three distinct ways.

Latency. In the paper's design-space exploration across four models and three workloads on a 128-GPU B200 cluster, AFD wins the user-interactivity axis in every panel.¹ The reason is that the attention-to-FFN ratio can be tuned until neither pool waits on the other, and the four-stage microbatch pipeline then overlaps compute with the dispatch and combine transfers. No aggregated layout has that degree of freedom.

Memory segmentation. Because the FFN pool holds the expert weights and the attention pool holds the KV cache, neither GPU has to hold both. In the paper's 1M-prefix case study the aggregated per-GPU requirement reaches roughly 298 GiB against a 180 GiB B200 budget, so no non-AFD layout is feasible at all, while the AFD requirement of roughly 165 GiB fits.⁴ This is the clearest case for AFD: it converts an infeasible deployment into a feasible one rather than merely a faster one.

Heterogeneous hardware. Once attention and FFN run on separate devices, they no longer have to run on the same kind of device. Memory-rich, bandwidth-heavy parts can serve attention while arithmetic-dense parts serve the experts. The paper positions AFD as the software primitive that makes emerging split-role inference hardware usable.⁵

Why it is not a default¶

The paper's abstract states that AFD "sustains around 4k tokens/s of system throughput on DeepSeek-V3.2 across chat, coding, and agentic-coding workloads, where non-AFD deployments are infeasible." Read the body before acting on that sentence. Section 1 states plainly that "AFD is not the best option when system throughput is the main target compared to the pure P/D disaggregation and chunked prefill," and the evaluation finds aggregated serving with chunked prefill wins most throughput panels by running 16 independent 8-GPU replicas and amortizing the prefill bubble across the fleet.¹ The infeasibility claim is also narrower than it sounds: the authors note that widening their search to TP-16 "unlocks a feasible disagg-standard layout for DeepSeek-V3.2 agentic coding," where a narrower search had returned nothing.¹ Non-AFD infeasibility was partly an artifact of search breadth.

The trade is concurrency. Each AFD replica consumes more GPUs than an aggregated replica, so a cluster fits fewer of them, and raw throughput usually comes from replica count. AFD buys latency and memory headroom, and pays in parallelism.

When to use it (and when not)¶

Use AFD when:

The per-GPU memory ceiling is the binding constraint. Long-prefix agentic serving where weights plus KV exceed device memory in any aggregated layout is the strongest case, and it is the one the paper demonstrates most convincingly.
Interactivity is the SLO that matters and throughput is a secondary target. Latency-sensitive serving is where AFD wins universally in the study.
Attention and FFN costs are badly mismatched for the model at hand. The further the ratio is from 1:1, the more an aggregated layout wastes one resource to satisfy the other.
The interconnect is full-duplex and fast. AFD adds two cross-pool transfers per layer. Without NVLink-class bandwidth between the pools, the added traffic dominates.

Do not use AFD when:

Aggregate throughput per GPU is the objective. Replicated aggregated workers with chunked prefill remain the stronger default.
The workload is short-context and retrieval-shaped. Large context with small ISL favours aggregated deployments, because frequent cross-node transfers cost more than the heterogeneity they resolve.
You need a supported production path today. See the production section: no mainstream engine ships AFD in a released version.
The model is dense rather than MoE. The dispatch and combine operators AFD repurposes exist because of expert routing. A dense FFN gives the technique far less to work with.

Architecture¶

The four stages and where the pipeline overlaps them:

sequenceDiagram
  participant A as "Attention GPUs"
  participant N as "Interconnect"
  participant F as "FFN GPUs"
  A->>A: "1. attention over KV cache, then route (top-k)"
  A->>N: "2. A2F dispatch (fan-out to every FFN rank)"
  N->>F: "tokens + routing metadata"
  F->>F: "3. local expert shard GEMMs (no collective)"
  F->>N: "4. F2A combine (fan-in, partial outputs)"
  N->>A: "partials summed, shared-expert output added"
  Note over A,F: "microbatch m+1 occupies stage 1 while m is in stage 3"

Under steady-state cross-layer pipelining the bottleneck stage processes all M microbatches back to back, while every other stage contributes only a one-time fill and drain cost amortized over L layers. That is the paper's Equation 1:

t_pipe = M * s_max + sum over stages i where s_i != s_max of (s_i / L)

M is chosen to match the effective pipeline depth: three microbatches on half-duplex links, four on full-duplex links where the return path can proceed concurrently on a dedicated channel.⁶

How to use it: pick the split, then check it three ways¶

The attention-to-FFN ratio is the one decision AFD forces, and the paper's rule is rate matching: allocate attention GPUs only as far as needed to match the FFN pool's output rate, parameterized by per-token attention cost and KV or state memory. Cheap, low-memory attention sustains a large FFN pool from a small attention slice; expensive attention shifts the ratio the other way.

The observed splits make the rule concrete. DeepSeek-V3.2 pairs MLA latent-KV compression with sparse attention, making per-token attention both cheap and small, and collapses to 2A+126F on the agentic workload and 16A+112F on chat. Qwen3-235B's dense GQA needs a wider slice at 8A+120F on agentic. GPT-OSS-120B's alternating sliding-window GQA sits in the cheap-attention regime on long prefixes, yet on chat a near-symmetric 16A+16F split is the one configuration where AFD wins throughput outright. Nemotron-3-Super-120B inverts the pattern entirely: its Mamba2 plus GQA hybrid must propagate a recurrent state across a 524k-token prefix, and the throughput-optimal layout is attention-heavy at 96A+32F.¹

This block models the three checks a candidate split has to pass: the pipeline latency it implies, the KV capacity it leaves, and the link congestion it creates. Run: python3 afd_model.py.

import numpy as np

# --- 1. Four-stage microbatch pipeline latency (paper Eq. 1) -----------------
# t_pipe = M * s_max + sum_{i : s_i != s_max} s_i / L
# s_i is the per-microbatch cost of stage i aggregated over all L layers.


def pipe_latency(stages: dict[str, float], M: int, L: int) -> float:
    s = np.array(list(stages.values()), dtype=float)
    assert M >= 1 and L >= 1, "M and L are counts"
    s_max = s.max()
    drain = s[s != s_max].sum() / L
    return M * s_max + drain


AFD_STAGES = {"attn": 0.90, "a2f": 0.35, "ffn": 1.40, "f2a": 0.35}  # ms/microbatch, all L layers

t4 = pipe_latency(AFD_STAGES, M=4, L=61)
print(f"[1] 4 stages {AFD_STAGES}, M=4, L=61 -> t_pipe = {t4:.4f} ms")
# Bottleneck is ffn (1.40). Steady state 4*1.40 = 5.60; drain (0.90+0.35+0.35)/61.
assert np.isclose(t4, 4 * 1.40 + (0.90 + 0.35 + 0.35) / 61), "Eq. 1 mismatch"

# M=1 degenerates to no pipelining: one microbatch, still pays the drain term.
t1 = pipe_latency(AFD_STAGES, M=1, L=61)
print(f"[1] M=1 (no overlap)          -> t_pipe = {t1:.4f} ms  (serial sum = "
      f"{sum(AFD_STAGES.values()):.4f} ms)")
assert t1 < sum(AFD_STAGES.values()), "pipelined M=1 must still beat the serial sum"

# Adversarial: the paper writes the drain sum over stages with s_i != s_max.
# When two stages TIE at the maximum, both are excluded and the drain shrinks.
TIED = {"attn": 1.40, "a2f": 0.35, "ffn": 1.40, "f2a": 0.35}
t_tie = pipe_latency(TIED, M=4, L=61)
print(f"[1] tie at s_max (attn==ffn)  -> t_pipe = {t_tie:.4f} ms  "
      f"(drain counts {int((np.array(list(TIED.values())) != 1.40).sum())} of 4 stages)")
assert np.isclose(t_tie, 4 * 1.40 + (0.35 + 0.35) / 61)
# Consequence: a tie makes the model OPTIMISTIC relative to the untied case,
# because the tied stage's fill/drain is dropped rather than charged once.
assert t_tie < t4, "tie-handling lowers modelled latency; treat as a model artifact"

# Half-duplex links use M=3, full-duplex M=4 (Section 3.2).
for M in (3, 4):
    print(f"[1] M={M} ({'half' if M == 3 else 'full'}-duplex) -> "
          f"{pipe_latency(AFD_STAGES, M=M, L=61):.4f} ms for {M} microbatches")

# --- 2. Rate-matched attention:FFN GPU split --------------------------------
# AFD sizes the two pools so neither starves: N_A * (1/a) == N_F * (1/f),
# i.e. N_A / N_F == a / f, where a and f are per-token GPU-seconds.


def rate_matched_split(a_cost: float, f_cost: float, n_gpus: int) -> tuple[int, int]:
    assert a_cost > 0 and f_cost > 0 and n_gpus >= 2
    n_a = max(1, round(n_gpus * a_cost / (a_cost + f_cost)))
    return n_a, n_gpus - n_a


print()
CASES = [
    ("DeepSeek-V3.2 agentic (MLA+DSA: cheap attn)", 0.016, 1.0),
    ("Qwen3-235B agentic (dense GQA)", 0.065, 1.0),
    ("GPT-OSS-120B chat (sliding-window GQA)", 1.0, 1.0),
    ("Nemotron-3 long ctx (Mamba2 state propagation)", 3.0, 1.0),
]
for name, a, f in CASES:
    na, nf = rate_matched_split(a, f, 128)
    print(f"[2] {name:47s} a/f={a / f:5.3f} -> {na:3d}A + {nf:3d}F "
          f"({100 * nf / 128:.0f}% of cluster on FFN)")

# The split is monotone in the attention cost ratio: cheaper attention always
# yields a smaller attention pool. This is the paper's stated rate-matching rule.
ratios = [0.01, 0.05, 0.2, 0.5, 1.0, 3.0, 9.0]
splits = [rate_matched_split(r, 1.0, 128)[0] for r in ratios]
assert all(b > a for a, b in zip(splits, splits[1:])), "attention share rises with attention cost"
print(f"[2] monotonicity over a/f={ratios}: N_A={splits} strictly increasing")

# Adversarial: rate matching alone cannot justify a split that cannot hold the
# KV cache. A 2A+126F layout is only valid if the KV working set fits 2 GPUs.
kv_bytes_per_token = 70_000  # DeepSeek-V3.2 MLA latent KV, order-of-magnitude
concurrent_tokens = 524_288
kv_gib = kv_bytes_per_token * concurrent_tokens / 2**30
print(f"[2] KV working set for one 524k-token request = {kv_gib:.1f} GiB; "
      f"2 attention B200s hold {2 * 180:.0f} GiB -> "
      f"{'fits' if kv_gib < 2 * 180 else 'DOES NOT FIT'}")
assert kv_gib < 2 * 180, "the 2A layout depends on MLA latent compression"

# --- 3. Per-GPU memory feasibility (Section 4.1.3) --------------------------
CAP = 180.0  # GiB usable HBM per B200
W_ATTN, W_EXPERT = 150.0, 2650.0  # GiB, FP8 weights for a large sparse MoE
ACT, BUF = 14.0, 9.0              # activations, NCCL buffers + runtime overhead


def kv_capacity(n_a: int, n_f: int) -> tuple[float, float]:
    """Cluster-wide KV capacity in GiB, aggregated vs AFD, for n_a+n_f GPUs.

    Aggregated: every GPU carries a shard of ALL weights, so KV shares what is
    left on all n GPUs. AFD: only the attention pool holds KV, but those GPUs
    have shed the expert weights entirely. AFD therefore trades a larger
    per-GPU KV headroom against a smaller number of GPUs holding KV.
    """
    n = n_a + n_f
    assert n_a >= 1 and n_f >= 1
    agg_head = CAP - (W_ATTN + W_EXPERT) / n - ACT - BUF
    attn_head = CAP - W_ATTN / n_a - ACT - BUF
    ffn_side = W_EXPERT / n_f + 0.3 * ACT + BUF
    agg_total = n * agg_head if agg_head > 0 else 0.0
    afd_total = n_a * attn_head if attn_head > 0 and ffn_side <= CAP else 0.0
    return agg_total, afd_total


print(f"[3] weights {W_ATTN + W_EXPERT:.0f} GiB, cap {CAP:.0f} GiB/GPU, 64-GPU replica")
for n_a, n_f in [(16, 48), (32, 32), (48, 16), (52, 12)]:
    agg_kv, afd_kv = kv_capacity(n_a, n_f)
    verdict = "AFD wins" if afd_kv > agg_kv else ("infeasible" if afd_kv == 0 else "AFD loses")
    print(f"[3] {n_a:3d}A+{n_f:3d}F -> aggregated KV {agg_kv:7.0f} GiB, "
          f"AFD KV {afd_kv:7.0f} GiB   {verdict}")

# The direction is NOT fixed by AFD; it is set by the split. FFN-heavy splits
# concentrate KV on too few GPUs and lose capacity outright.
agg_kv, afd_kv = kv_capacity(16, 48)
assert afd_kv < agg_kv, "FFN-heavy split concentrates KV and reduces total capacity"
agg_kv, afd_kv = kv_capacity(48, 16)
assert afd_kv > agg_kv, "attention-heavy split sheds expert weights and gains capacity"

# Adversarial: push the FFN pool too small and the expert shard alone exceeds the
# cap, so the layout is infeasible no matter how much KV headroom attention has.
ffn_only = W_EXPERT / 12 + 0.3 * ACT + BUF
print(f"[3] 52A+12F ffn side = {ffn_only:.1f} GiB vs {CAP:.0f} cap -> "
      f"{'FEASIBLE' if ffn_only <= CAP else 'INFEASIBLE'}")
assert kv_capacity(52, 12)[1] == 0.0, "expert shard must fit the FFN pool first"

# The paper reports 298 GiB shared versus 165 GiB under AFD against a 180 GiB
# cap for its 1M-prefix Qwen3-235B case, but does not break out the W/A/K/N/O
# terms, so that specific reduction cannot be re-derived from the paper alone.
PAPER_SHARED, PAPER_AFD = 298.0, 165.0
print(f"[3] paper's reported case: shared {PAPER_SHARED:.0f} GiB > {CAP:.0f} cap > "
      f"AFD {PAPER_AFD:.0f} GiB (terms not broken out; not re-derivable here)")
assert PAPER_SHARED > CAP > PAPER_AFD

# --- 4. Bipartite dispatch/combine ingress load (Section 3.1) ---------------
# All-pairs AFD builds M*N NCCL pair-groups. A2F fans out (FFN ingress is the
# hot side); F2A fans in (attention ingress is the hot side).
print()


def ingress(M_attn: int, N_ffn: int, payload_mb: float) -> tuple[float, float, int]:
    assert M_attn >= 1 and N_ffn >= 1
    pairs = M_attn * N_ffn
    ffn_ingress = M_attn * payload_mb      # each FFN rank receives from every attn rank
    attn_ingress = N_ffn * payload_mb      # each attn rank receives from every FFN rank
    return ffn_ingress, attn_ingress, pairs


for (ma, nf) in [(16, 16), (2, 126), (96, 32)]:
    fi, ai, pairs = ingress(ma, nf, payload_mb=4.0)
    hot = "FFN ingress" if fi > ai else ("attention ingress" if ai > fi else "balanced")
    print(f"[4] {ma:3d}A+{nf:3d}F -> {pairs:5d} NCCL pair-groups, "
          f"FFN ingress {fi:6.1f} MB/layer, attn ingress {ai:6.1f} MB/layer, hot side: {hot}")

fi, ai, _ = ingress(16, 16, 4.0)
assert fi == ai, "M==N is the only balanced case"
fi, ai, _ = ingress(2, 126, 4.0)
assert ai > fi, "M<N congests the attention (combine) side"
fi, ai, _ = ingress(96, 32, 4.0)
assert fi > ai, "M>N congests the FFN (dispatch) side"

# Pair-group count grows as the product, not the sum. This is the scaling wall.
for n in (8, 16, 64, 128):
    a, f = n // 2, n - n // 2
    print(f"[4] symmetric split on {n:3d} GPUs -> {a * f:5d} NCCL communicators to bootstrap")
assert (64 * 64) == ingress(64, 64, 1.0)[2] == 4096

# --- 5. Placement: which tier carries which transfer (Section 6.2) ----------
print()
BW_NVLINK, BW_IB = 450.0, 25.0  # GB/s, paper's modelled link rates
print(f"[5] KV transfer speed-up from keeping a P/D pair in one NVLink domain: "
      f"{BW_NVLINK / BW_IB:.0f}x")
assert np.isclose(BW_NVLINK / BW_IB, 18.0), "paper reports 18x; it is exactly the BW ratio"
# The frequency argument: per-layer A2F/F2A traffic dwarfs the once-per-request
# KV transfer, so A2F/F2A gets the fast tier.
L, layers_per_req, kv_per_req = 61, 2, 1
print(f"[5] per request: {L * layers_per_req} AFD transfers vs {kv_per_req} KV transfer "
      f"-> bind AFD traffic to NVLink, defer KV to InfiniBand")
assert L * layers_per_req > kv_per_req * 100

print("\nAll assertions passed.")

Executed output:

[1] 4 stages {'attn': 0.9, 'a2f': 0.35, 'ffn': 1.4, 'f2a': 0.35}, M=4, L=61 -> t_pipe = 5.6262 ms
[1] M=1 (no overlap)          -> t_pipe = 1.4262 ms  (serial sum = 3.0000 ms)
[1] tie at s_max (attn==ffn)  -> t_pipe = 5.6115 ms  (drain counts 2 of 4 stages)
[1] M=3 (half-duplex) -> 4.2262 ms for 3 microbatches
[1] M=4 (full-duplex) -> 5.6262 ms for 4 microbatches

[2] DeepSeek-V3.2 agentic (MLA+DSA: cheap attn)     a/f=0.016 ->   2A + 126F (98% of cluster on FFN)
[2] Qwen3-235B agentic (dense GQA)                  a/f=0.065 ->   8A + 120F (94% of cluster on FFN)
[2] GPT-OSS-120B chat (sliding-window GQA)          a/f=1.000 ->  64A +  64F (50% of cluster on FFN)
[2] Nemotron-3 long ctx (Mamba2 state propagation)  a/f=3.000 ->  96A +  32F (25% of cluster on FFN)
[2] monotonicity over a/f=[0.01, 0.05, 0.2, 0.5, 1.0, 3.0, 9.0]: N_A=[1, 6, 21, 43, 64, 96, 115] strictly increasing
[2] KV working set for one 524k-token request = 34.2 GiB; 2 attention B200s hold 360 GiB -> fits
[3] weights 2800 GiB, cap 180 GiB/GPU, 64-GPU replica
[3]  16A+ 48F -> aggregated KV    7248 GiB, AFD KV    2362 GiB   AFD loses
[3]  32A+ 32F -> aggregated KV    7248 GiB, AFD KV    4874 GiB   AFD loses
[3]  48A+ 16F -> aggregated KV    7248 GiB, AFD KV    7386 GiB   AFD wins
[3]  52A+ 12F -> aggregated KV    7248 GiB, AFD KV       0 GiB   infeasible
[3] 52A+12F ffn side = 234.0 GiB vs 180 cap -> INFEASIBLE
[3] paper's reported case: shared 298 GiB > 180 cap > AFD 165 GiB (terms not broken out; not re-derivable here)

[4]  16A+ 16F ->   256 NCCL pair-groups, FFN ingress   64.0 MB/layer, attn ingress   64.0 MB/layer, hot side: balanced
[4]   2A+126F ->   252 NCCL pair-groups, FFN ingress    8.0 MB/layer, attn ingress  504.0 MB/layer, hot side: attention ingress
[4]  96A+ 32F ->  3072 NCCL pair-groups, FFN ingress  384.0 MB/layer, attn ingress  128.0 MB/layer, hot side: FFN ingress
[4] symmetric split on   8 GPUs ->    16 NCCL communicators to bootstrap
[4] symmetric split on  16 GPUs ->    64 NCCL communicators to bootstrap
[4] symmetric split on  64 GPUs ->  1024 NCCL communicators to bootstrap
[4] symmetric split on 128 GPUs ->  4096 NCCL communicators to bootstrap

[5] KV transfer speed-up from keeping a P/D pair in one NVLink domain: 18x
[5] per request: 122 AFD transfers vs 1 KV transfer -> bind AFD traffic to NVLink, defer KV to InfiniBand

All assertions passed.

Three results in that output are worth carrying into a design review.

Section 3 is this page's own model, and it contradicts a naive reading of the paper. AFD's memory effect is not a one-way win. Concentrating the KV cache on a smaller attention pool cuts against the saving from shedding expert weights, so an FFN-heavy split can leave the cluster with less total KV capacity than an aggregated layout. In the modelled 64-GPU replica, 16A+48F loses badly, 48A+16F wins by a small margin, and 52A+12F is infeasible because the expert shard alone overflows the FFN GPUs. The memory benefit is a property of the split, not of AFD.

The tie case in Section 1 is a modelling artifact worth knowing about. Equation 1 sums the fill and drain term over stages whose cost differs from the maximum. When two stages tie at the bottleneck, both drop out of that sum, so the model reports a slightly lower latency than an almost-identical unbalanced configuration. Do not read small differences between near-balanced configurations as real.

Communicator count grows as the product of the pool sizes. A symmetric split on 128 GPUs needs 4,096 NCCL communicators bootstrapped through per-pair Gloo rendezvous on unique TCP ports. That is a startup-time and file-descriptor problem long before it is a bandwidth problem, and it is the practical reason all-pairs AFD does not simply scale to arbitrary cluster sizes.

How to develop with it¶

There is no supported API to build against, so development means working from the prototype's structure. Four decisions define a correct implementation:

Move the router before the transport. Routing on the attention side is what eliminates FFN-to-FFN collectives. If the router stays on the FFN side, the dispatch becomes an all-gather and the design collapses back into ordinary expert parallelism with an extra hop.
Keep shared experts with attention. Shared experts run for every token, so co-locating them with attention avoids sending that traffic across the link at all.
Bypass the modular MoE kernel path. Calling fused_experts directly with an expert_map is what makes the FFN side collective-free. Entering through FusedMoEModularKernel.forward reintroduces the very collectives the architecture removes.
Make dense and sparse transport bit-identical. The prototype's two modes must agree exactly given identical routing. Test that equivalence first; it is the cheapest correctness signal available and it catches routing-metadata bugs that otherwise surface as silent quality regressions.

Validate against a non-AFD baseline on logits, not on benchmark scores. An MoE routing bug shifts a small fraction of tokens to the wrong expert, which barely moves an aggregate score while corrupting individual generations.

How to run it in production¶

State the honest answer first: as of 2026-07-30 you cannot, on a released mainstream engine. Verified directly against the repositories:

Path	Status (checked 2026-07-30)
vLLM PR #29772, the paper's starting point	Closed, never merged
vLLM PR #33961 "Elastic AFD"	Open, still a draft (created 2026-02-06)
SGLang RFC #10900, Attention-FFN Disaggregation	Open, not merged
StepFun StepMesh (AFD transport library)	Apache-2.0, C++, last push 2026-01-28

Running AFD today means carrying a fork. Plan for that cost explicitly, including rebasing against an engine whose MoE kernel entry points move frequently.

When you do deploy, placement is the decision that matters most, and it follows a frequency argument. Per-layer dispatch and combine traffic occurs 2 * L times per request; the prefill-to-decode KV transfer occurs once. Bind the frequent traffic to the fastest tier:

Put the attention and FFN pools of one replica inside the same scale-up domain (intra-node NVLink), so A2F and F2A never touch the scale-out fabric.
Defer the KV-cache transfer to the scale-out fabric (InfiniBand), since it is paid once per request.
Pair each prefill worker with its decode worker on the same node rather than segregating all prefill nodes from all decode nodes. In the paper's 2A2F with 2P2D layout across two 8-GPU nodes, paired placement keeps KV transfers on NVLink at a modelled 450 GB/s instead of crossing InfiniBand at 25 GB/s, an 18x improvement that is exactly the bandwidth ratio.⁷

The generalization holds for any tiered topology: assign communication patterns to link tiers in order of frequency, fastest tier to most frequent pattern.

Gate the deployment on SLOs, not throughput. The paper's own constraint set is a reasonable starting template: TTFT under 50 ms for chat, 100 ms for coding, and 150 ms for agentic coding, with a TPOT cap of 15 ms.¹ Track inference serving SLOs and treat any configuration that misses them as infeasible rather than merely slow.

How to maintain it¶

Re-derive the split when the workload moves. The optimal ratio is a function of ISL, OSL, prefix length, and load, not a constant of the model. A shift from chat traffic to agentic traffic moved DeepSeek-V3.2's split from 16A+112F to 2A+126F in the study.
Re-derive it when the attention mechanism changes. Swapping GQA for MLA, adding sparse attention, or introducing a recurrent mixer moves the ratio by an order of magnitude. Nemotron-3's Mamba2 mixer flips the layout attention-heavy where every dense-attention model in the study sits FFN-heavy.
Watch pool-level utilization as the primary health signal. A persistently idle pool means the split has drifted from rate matching. This is the AFD-specific analogue of the prefill/decode imbalance covered in rate matching.
Re-check engine support each release. The status table above is a point-in-time observation of fast-moving upstream work; confirm it rather than inheriting it.
Pin the model and engine revision together. The prototype targets vLLM v0.16.0rc2, and the FusedMoE entry points it patches are internal APIs with no stability guarantee.

Failure modes¶

Communicator bootstrap storm. N_A * N_F NCCL pair-groups built through per-pair Gloo rendezvous on unique TCP ports. Symptoms are long startup, port exhaustion, and file-descriptor limits. Cap the split product or move to a hierarchical topology.
Ingress congestion on the wrong side. N_A > N_F congests FFN ingress on dispatch; N_A < N_F congests attention ingress on combine. Only N_A == N_F is balanced. A split chosen purely from compute rate matching can put the hot side on the pool with less network headroom.
Split that rate-matches but does not fit. Compute balance and memory feasibility are separate constraints. Check that the expert shard fits the FFN pool and the KV working set fits the attention pool before accepting a ratio.
AFD applied for throughput. The most likely misuse. Aggregated replicas win most throughput panels in the source study; reaching for AFD to raise tokens per second per GPU usually loses.
Reading simulated results as measured ones. The cluster-scale figures are design-space-exploration estimates. Treat them as a ranking of configurations, not as a performance guarantee for your hardware.
Trusting the abstract's infeasibility claim. Non-AFD infeasibility in the study was partly an artifact of a narrow parallelism search. Widen your own search before concluding no aggregated layout works.
Quality regression from routing bugs. Attention-side routing moves the router across a process boundary. Metadata that disagrees with the hidden states it accompanies produces wrong-expert execution that aggregate benchmarks hide.

Open questions and validation¶

The study evaluates B200 only. Whether the conclusions transfer to hardware with a different compute-to-bandwidth ratio is untested, and the authors name this as future work.
The prototype verifies functional correctness, not performance. No end-to-end measured throughput for the all-pairs bipartite path has been published.
The paper's Section 2.1 expands AFD once as "asynchronous function decomposition," which contradicts the definition used everywhere else in the same paper. Read it as a drafting error, not a second technique.
Whether all-pairs pairing is the right topology at scale is open. The communicator count argues for hierarchy, and the open "Elastic AFD" work suggests upstream is exploring alternatives.

References¶

Wu et al., "How Far Can Disaggregation Go? A Design-Space Exploration of Attention-FFN Disaggregation for Efficient MoE LLM Serving", arXiv 2605.28302: https://arxiv.org/abs/2605.28302
Zhu et al., "MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism", arXiv 2504.02263: https://arxiv.org/abs/2504.02263
Xu et al., "AIConfigurator: Lightning-fast configuration optimization for multi-framework LLM serving", arXiv 2601.06288: https://arxiv.org/abs/2601.06288
ASTRA-sim network simulator: https://github.com/astra-sim/astra-sim
StepFun StepMesh, communication library for attention-FFN disaggregation: https://github.com/stepfun-ai/StepMesh
vLLM PR #29772, "[Feature] AFD basic implemetation" (closed): https://github.com/vllm-project/vllm/pull/29772
vLLM PR #33961, "Elastic AFD" (open draft): https://github.com/vllm-project/vllm/pull/33961
SGLang RFC #10900, Attention-FFN Disaggregation: https://github.com/sgl-project/sglang/pull/10900

Wu et al., arXiv 2605.28302. Section 4.1.2 states the cluster-scale results "are model-based DSE estimates that combine backend cost measurements with AstraSim communication modeling", on 128 B200 SXM GPUs with TensorRT-LLM as the cost backend. The SLO set (TTFT under 50/100/150 ms for chat/coding/agentic coding, TPOT cap 15 ms) is from the Figure 2 caption. Key Takeaway 1 credits aggregated serving for throughput; Key Takeaway 2 credits AFD for interactivity. The TP-16 remark on unlocking a feasible non-AFD layout is in the same section. ↩↩↩↩↩↩
Wu et al., arXiv 2605.28302, Appendix 6.1. The prototype is built on vLLM v0.16.0rc2 and is described as being "used for correctness and implementation guidance, while the cluster-scale results use measured backend costs and AstraSim communication modeling." ↩
The paper cites StepMesh as the production precedent for the same point-to-point send/receive pattern. Repository metadata checked 2026-07-30: Apache-2.0, C++, last push 2026-01-28. ↩
Wu et al., arXiv 2605.28302, Section 4.1.3. Reported as roughly 298 GiB shared versus roughly 165 GiB under AFD against a 180 GiB B200 capacity. The paper does not break out the individual weight, activation, KV, NCCL-buffer, and overhead terms, so the reduction cannot be independently re-derived from the paper. Throughput in the same section: Agg+AFD with M=4 reaches 1346.5 tok/s at 64 GPUs and 2693.0 tok/s at 128 GPUs on the ISL 500k plus OSL 10k workload, using 2 and 4 replicas of a 28A+4F layout respectively. ↩
Wu et al., arXiv 2605.28302, Section 2.2, which motivates AFD by "the emergence of disaggregated architectures with heterogeneous compute units within a node". The specific accelerators it names are vendor announcements cited by the paper, not products validated here. ↩
Wu et al., arXiv 2605.28302, Section 3.2 and Equation 1. The number of microbatches M is "chosen to match the effective pipeline depth", three for half-duplex links and four for full-duplex links. ↩
Wu et al., arXiv 2605.28302, Appendix 6.2. The comparison is a 2A2F configuration in a 2P2D layout across two 8-GPU nodes, with modelled link rates of 450 GB/s for NVLink and 25 GB/s for InfiniBand. ↩