Markdown

Continuous batching in transformers¶

Scope: serving and batch-generating with transformers' own continuous batching engine (model.generate_batch, ContinuousBatchingManager, transformers serve --continuous-batching), covering the paged KV cache and its two reserved blocks, the fifo and prefill_first schedulers, the safety_margin admission reserve, the offload-then-soft-reset response to KV pressure, the decode fast path and its silent fallback, and the asynchronous CUDA-streams loop. For the engine-agnostic scheduler theory see continuous batching internals; for running a transformers model inside vLLM instead, see the vLLM Transformers modeling backend.

Code on this page comes in two kinds. The three numpy blocks are executed: they run on a stock python3 with numpy, assert every number quoted around them, and include boundary and must-raise cases. The transformers snippets need a GPU and the library, are not executed here, and are labelled as reference templates pinned to transformers==5.13.1. Symbols and defaults were read off main on 2026-07-13; re-verify against your installed version, because several of these defaults have moved.

What it is¶

transformers ships a continuous batching engine inside the library itself, in src/transformers/generation/continuous_batching/. It is not a wrapper around vLLM. It is a paged-KV-cache serving loop with its own scheduler, block allocator, prefix cache, CUDA-graph capture, and CPU-offload tier, reachable from any causal LM through the ContinuousMixin methods.

The public surface is small. ContinuousMixin exposes three entry points that form a strict hierarchy:

Entry point	What it does	Use it for
`model.generate_batch(inputs, ...)`	Blocks until every prompt is done, returns `dict[str, GenerationOutput]`	Offline batch generation, evals, RL rollouts
`model.continuous_batching_context_manager(...)`	Yields a started manager, stops (and destroys) it on exit	Scoped streaming or interactive use
`model.init_continuous_batching(...)`	Returns a `ContinuousBatchingManager` you start and stop yourself	Long-lived servers

generate_batch wraps the context manager, which wraps init_continuous_batching. The manager runs a background thread; requests are submitted with add_request / add_requests, cancelled with cancel_request, and collected with get_result, by iterating the manager, or through request_id_iter for token-by-token streaming. There is no request() method, a plausible-looking name that does not exist.

The package's __all__ is exactly ContinuousBatchingManager, ContinuousMixin, FIFOScheduler, PagedAttentionCache, PrefillFirstScheduler, RequestState, RequestStatus, Scheduler. Everything else is configured through ContinuousBatchingConfig.

Continuous batching requires a paged attention backend. If a model is loaded with a plain implementation, the manager rewrites the implementation string to paged|<impl> when it starts. Supported combinations are paged|flash_attention_2, paged|sdpa, and paged|eager.

Why use it¶

Continuous batching changes request admission and retirement at iteration boundaries. Ragged or paged attention is a separate optimization that removes rectangular padding inside each iteration; the two are usually deployed together. The padding arithmetic is worth deriving because the canonical explainer for this engine misstates its growth rate.

When a fresh n-token prompt joins B-1 requests that are decoding one token each, a padded (rectangular) batch must pad every short row out to n. The Hugging Face post derives the exact cost, (n-1)(B-1) wasted slots, and then says the cost "grows quadratically with both batch size and prompt length". It does not. (n-1)(B-1) is linear in each variable separately and bilinear jointly: doubling the batch size roughly doubles the waste, it does not quadruple it. The distinction matters because it changes what you tune. The block below reproduces the post's own 693-slot example and then proves the growth is linear, by showing the second difference in each variable is exactly zero.

# Runnable on system python3 (numpy). Core claim: padded (dynamic) batching wastes
# (n-1)(B-1) token slots when one prompt of n tokens joins B-1 decoding requests, and that
# waste is LINEAR in each variable, not quadratic. Ragged batching wastes nothing.
import numpy as np


def padded_slots(batch: list[int]) -> int:
    """A padded batch is a rectangle: rows x longest row."""
    assert batch and min(batch) > 0, "every row needs at least one token"
    return len(batch) * max(batch)


def useful_slots(batch: list[int]) -> int:
    """Ragged/packed batching stores exactly the real tokens: no rectangle, no padding."""
    return sum(batch)


def padding_waste(B: int, n: int) -> int:
    """Waste when a fresh n-token prefill joins B-1 decodes (1 token each)."""
    assert B >= 1 and n >= 1
    batch = [n] + [1] * (B - 1)
    return padded_slots(batch) - useful_slots(batch)


# 1. The blog's own example, reproduced exactly: B=8, n=100 -> 99 * 7 = 693 wasted slots.
assert padding_waste(8, 100) == 693
assert padding_waste(8, 100) == (100 - 1) * (8 - 1)

# 2. The closed form holds for every (B, n), against the brute-force rectangle.
for B in range(1, 40):
    for n in range(1, 40):
        assert padding_waste(B, n) == (n - 1) * (B - 1)

# 3. Ragged batching (what continuous batching actually does) wastes zero slots, by
#    construction: there is no rectangle to pad into.
assert useful_slots([100] + [1] * 7) == 107
assert padded_slots([100] + [1] * 7) == 800          # 8 rows x 100 wide
assert 800 - 107 == 693                              # the same 693, seen as a rectangle

# 4. NOT quadratic. Double B alone -> waste grows ~2x (not 4x). Double n alone -> ~2x.
#    A function quadratic in B would give exactly 4x.
r_B = padding_waste(16, 100) / padding_waste(8, 100)
r_n = padding_waste(8, 200) / padding_waste(8, 100)
assert 2.0 <= r_B < 2.5, r_B                         # 15/7 = 2.142..., nowhere near 4
assert 2.0 <= r_n < 2.5, r_n                         # 199/99 = 2.010...
r_both = padding_waste(16, 200) / padding_waste(8, 100)
assert 4.0 <= r_both < 4.5, r_both                   # bilinear: doubling BOTH gives ~4x

# 5. Exact proof of linearity: the second difference of a linear function is exactly 0.
#    (For a quadratic in B, the second difference would be a non-zero constant.)
second_diff_B = [padding_waste(B + 2, 100) - 2 * padding_waste(B + 1, 100) + padding_waste(B, 100)
                 for B in range(1, 20)]
second_diff_n = [padding_waste(8, n + 2) - 2 * padding_waste(8, n + 1) + padding_waste(8, n)
                 for n in range(1, 20)]
assert set(second_diff_B) == {0}, second_diff_B
assert set(second_diff_n) == {0}, second_diff_n
# The MIXED second difference is the non-zero one: that is what "bilinear" means.
mixed = (padding_waste(9, 101) - padding_waste(9, 100)
         - padding_waste(8, 101) + padding_waste(8, 100))
assert mixed == 1, mixed

# 6. Boundaries: a batch of one, or a one-token prompt, wastes nothing. And a zero-length
#    row is not a batch: it must raise rather than silently pad against a phantom row.
assert padding_waste(1, 100) == 0                    # nothing to pad against
assert padding_waste(8, 1) == 0                      # prompt already one token wide
raised = False
try:
    padded_slots([100, 0, 1])
except AssertionError:
    raised = True
assert raised, "a zero-length row must raise, not be padded silently"

print("padding waste OK")
print(f"  B=8, n=100     -> {padding_waste(8, 100)} wasted slots ({padded_slots([100] + [1]*7)} padded"
      f" vs {useful_slots([100] + [1]*7)} real)")
print(f"  double B       -> x{r_B:.3f}   (quadratic in B would be x4.000)")
print(f"  double n       -> x{r_n:.3f}   (quadratic in n would be x4.000)")
print(f"  double both    -> x{r_both:.3f}   (bilinear: this is the ~4x)")
print(f"  2nd difference in B = {set(second_diff_B)}, in n = {set(second_diff_n)}, mixed = {mixed}")

Executed output:

padding waste OK
  B=8, n=100     -> 693 wasted slots (800 padded vs 107 real)
  double B       -> x2.143   (quadratic in B would be x4.000)
  double n       -> x2.010   (quadratic in n would be x4.000)
  double both    -> x4.307   (bilinear: this is the ~4x)
  2nd difference in B = {0}, in n = {0}, mixed = 1

Ragged batching removes 693 slots of rectangular padding in this example. CUDA graphs do not inherently require every row to be padded to the model's maximum context: production engines capture shape buckets and pass sequence lengths or block tables to paged kernels. Continuous batching's defining gain is iteration-level admission and retirement; ragged or paged attention removes the per-step padding quantified here.

Beyond throughput, the reasons to use this engine rather than a dedicated one are specific:

One library for training, evaluation, rollout, and serving. RL rollouts want return_logprobs=True from the same model definition used to train. Running rollouts on a separate engine introduces train/serve skew that surfaces as a misbehaving reward curve, not a crash.
Day-zero coverage. A model merged into transformers is servable with continuous batching immediately, with no port to a serving engine.
A short path from research code to a throughput-shaped loop. generate_batch is a drop-in for a padded generate loop over a list of prompts.

When to use it (and when not)¶

Situation	Verdict
Offline batch generation, evals, RL rollouts on a model already in `transformers`	Use it
Rollouts that need per-token logprobs from the training model definition	Use it (`return_logprobs=True`)
A model with no vLLM or SGLang implementation yet	Use it, or use the vLLM Transformers backend
A latency-critical, high-QPS production endpoint	Prefer a dedicated engine; see below
You need disaggregated prefill/decode, speculative decoding, or structured output	Not available here; use vLLM or SGLang
Sliding-window model with heavy prefix reuse	Works, but prefix caching and the decode fast path are both disabled automatically; expect no reuse and varlen decode

The honest comparison against vLLM and SGLang is architectural, not numerical. There is no citable, reproducible throughput ratio between transformers continuous batching and vLLM: published third-party claims range from "84% of vLLM" to multiples in either direction, none with a method you can re-run. This page therefore states no such ratio, and neither should a capacity plan built on it. What can be compared is the shape of the two engines: transformers exposes an explicit admission reserve (safety_margin) that vLLM does not, prefers CPU swap over recompute under KV pressure when it is configured to, and lacks disaggregation and speculative decoding entirely. Benchmark your own workload before choosing.

Architecture¶

The manager owns a background thread. The scheduler builds one batch per step under three budgets, the paged cache hands out blocks, and the offloading manager is the pressure-release valve.

flowchart TD
  A["add_request / add_requests"] --> Q["waiting queue (PENDING)"]
  Q --> S{"scheduler: fifo or prefill_first"}
  S -->|"below reserve: hold additional prefills<br/>(allow one if the step would be empty)"| Q
  S -->|"admit under token + cache + request budgets"| P["PREFILLING (chunked)"]
  P --> D["DECODING (1 token / step)"]
  D --> D
  D -->|"stop condition"| F["FINISHED: free blocks"]
  S --> C["PagedAttentionCache: num_blocks + 2"]
  C -->|"block table"| K{"batch is decode-only?"}
  K -->|"yes, and FA2/FA3 on CUDA"| FP["decode fast path: flash_attn_with_kvcache"]
  K -->|"no, or unsupported backend"| VL["varlen path: flash_attn_varlen_func"]
  D -->|"cannot allocate next block"| O["offloading manager"]
  O -->|"cpu_offload_space > 0 and it fits"| CPU["pinned CPU swap, resume later"]
  O -->|"otherwise"| SR["soft reset: append generated tokens to prompt, re-queue"]
  CPU --> Q
  SR --> Q

The paged cache, its two reserved blocks, and the block-size trade¶

The cache has a three-level hierarchy. A page is the key (or value) state for one token in one layer. A block is block_size pages and is the unit of allocation. Cache tensors are the physical support, one per layer in a layer group.

The allocated tensor is num_blocks + 2 blocks long. The two extra blocks are a padding zone no block manager ever hands out: the first holds the read-trash index (where padding tokens read from) and, one slot later, the sliding-window sentinel index; the second is the write-trash block (where padding tokens can safely write).

Three constants, taken as constants: _min_block_size = 4, so block_size below 4 raises a ValueError; the block_size default is 256; and the reserved zone is exactly 2 blocks. Resist the upstream doc's causal story for the first one. The architecture doc says the two-block reservation "requires block_size to be at least 4", but that does not follow from the code: the sentinel sits one slot after the read trash, so the bookkeeping needs only block_size >= 2, and the source comment on the sentinel line says as much (# since block size >= 4 >= 2, this is safe), treating 4 as a given rather than as the thing being derived. The one rationale the source does give is for the default, not the minimum: # Default used to be 32, now it's 256 to be compatible with flash_with_kvcache. So the minimum of 4 is a constant with no stated derivation, and the default of 256 exists for the decode kernel. Trust the constants, not the prose around them.

The block below builds that layout, checks the boundary arithmetic, proves block_size < 4 raises, and prices the block-size trade-off. It also corrects a second upstream doc claim: the docs say small blocks "spend a large fraction of memory on this fixed overhead", but the reserved zone is two blocks, not a fixed number of bytes, so its cost shrinks as block_size shrinks. The real cost of a small block is the per-request block table, which grows.

# Runnable on system python3 (numpy). The transformers paged cache: a page is the K (or V)
# state of ONE token in ONE layer; a block is block_size pages and is the allocation unit;
# the cache allocates num_blocks + 2 blocks, the last two being a padding/sentinel zone no
# block manager ever hands out. block_size < 4 is rejected.
import numpy as np

MIN_BLOCK_SIZE = 4          # PagedAttentionCache._min_block_size
RESERVED_BLOCKS = 2         # read-trash + sentinel, and write-trash


def make_cache(num_blocks: int, block_size: int, kv_heads: int, head_dim: int) -> np.ndarray:
    """Physical cache tensor for one layer: [(num_blocks + 2) * block_size, kv_heads, head_dim]."""
    if block_size < MIN_BLOCK_SIZE:
        raise ValueError(f"Block size must be at least {MIN_BLOCK_SIZE}, but got {block_size}")
    return np.zeros(((num_blocks + RESERVED_BLOCKS) * block_size, kv_heads, head_dim), dtype=np.float16)


def blocks_for(seq_len: int, block_size: int) -> int:
    """Blocks a sequence occupies: ceil-div. The last block is usually partly filled."""
    assert seq_len >= 0 and block_size >= MIN_BLOCK_SIZE
    return (seq_len + block_size - 1) // block_size


def internal_waste(seq_len: int, block_size: int) -> int:
    """Token slots paid for but unused, i.e. the tail of the last block."""
    return blocks_for(seq_len, block_size) * block_size - seq_len


BS = 256                    # ContinuousBatchingConfig.block_size default
cache = make_cache(num_blocks=64, block_size=BS, kv_heads=8, head_dim=128)

# 1. The reserved zone is real memory: the tensor is 2 blocks longer than the allocatable pool.
assert cache.shape[0] == (64 + 2) * BS
read_trash, sentinel, write_trash = 64 * BS, 64 * BS + 1, (64 + 1) * BS
assert sentinel == read_trash + 1                    # both live in the first reserved block
assert write_trash == read_trash + BS                # the second reserved block
assert max(read_trash, sentinel, write_trash) < cache.shape[0]   # in bounds, never allocated

# 2. Exact-multiple boundary: a full block wastes nothing, one token more costs a whole block.
assert blocks_for(BS, BS) == 1 and internal_waste(BS, BS) == 0
assert blocks_for(BS + 1, BS) == 2 and internal_waste(BS + 1, BS) == BS - 1
assert blocks_for(0, BS) == 0

# 3. Internal fragmentation is bounded by one block per sequence, never more.
lengths = [1, 255, 256, 257, 512, 1000, 4096, 8191]
assert all(internal_waste(L, BS) < BS for L in lengths)
assert max(internal_waste(L, BS) for L in lengths) == 255     # worst case: block_size - 1

# 4. Adversarial: block_size below the minimum must RAISE. _min_block_size = 4 is a bare
#    constant: the sentinel needs only block_size >= 2 (the source comment on that line reads
#    "since block size >= 4 >= 2, this is safe", i.e. it ASSUMES 4, it does not derive it), and
#    the 256 default is attributed to flash_with_kvcache compatibility, not to the reserved
#    zone. The upstream doc's "the reservation requires block_size >= 4" is not in the code.
for bad in (0, 1, 2, 3):
    raised = False
    try:
        make_cache(num_blocks=8, block_size=bad, kv_heads=8, head_dim=128)
    except ValueError:
        raised = True
    assert raised, f"block_size={bad} must raise"
make_cache(num_blocks=8, block_size=MIN_BLOCK_SIZE, kv_heads=8, head_dim=128)   # 4 is allowed

# 5. The block-size tradeoff, priced. For a FIXED KV budget, the reserved zone costs
#    2 * block_size pages, so it gets CHEAPER as blocks shrink, while the per-request block
#    table (max_blocks_per_request entries) gets more expensive. The upstream doc says small
#    blocks "spend a large fraction of memory on this fixed overhead"; the reserved zone is
#    two BLOCKS, not a fixed number of bytes, so as written that is backwards.
PAGE_BYTES = 2 * 8 * 128 * 2                         # K + V, kv_heads * head_dim * fp16
BUDGET = 4 * 1024**3                                 # 4 GiB of KV per layer
rows = []
for bs in (4, 16, 64, 256, 1024):
    num_blocks = BUDGET // (bs * PAGE_BYTES) - RESERVED_BLOCKS
    reserved_frac = RESERVED_BLOCKS / (num_blocks + RESERVED_BLOCKS)
    table_entries = blocks_for(8192, bs)             # block-table rows for an 8k sequence
    frag = max(internal_waste(L, bs) for L in range(1, 2 * bs + 1))   # attained worst case
    assert frag == bs - 1                            # the bound is tight, not just an upper bound
    rows.append((bs, num_blocks, reserved_frac, table_entries, frag))

# Reserved-zone fraction SHRINKS as block_size shrinks; block table and fragmentation GROW.
assert [r[2] for r in rows] == sorted(r[2] for r in rows), "reserved fraction must rise with block_size"
assert rows[0][2] < rows[-1][2]
assert rows[0][3] > rows[-1][3], "small blocks need a bigger block table"
assert rows[0][4] < rows[-1][4], "small blocks waste less on the last-block tail"

# 6. The default block table addresses exactly 8192 tokens. FALLBACK_DEFAULTS sets
#    max_blocks_per_request = 32 when no workload hint is given, and block_size is 256:
#    32 * 256 = 8192. A request longer than that needs a hint (max_prompt_length +
#    max_generated_length) or an explicit max_blocks_per_request, or the block table is
#    too small to address its KV.
FALLBACK_MAX_BLOCKS_PER_REQUEST = 32
assert FALLBACK_MAX_BLOCKS_PER_REQUEST * BS == 8192
assert blocks_for(8192, BS) == FALLBACK_MAX_BLOCKS_PER_REQUEST         # exactly fits
assert blocks_for(8193, BS) > FALLBACK_MAX_BLOCKS_PER_REQUEST          # one token over: does not

# What a hint would give instead, per resolve_using_hints: ceil(max_seq / block_size) + 1,
# rounded up to an even number of blocks.
def blocks_from_hint(max_prompt: int, max_generated: int, block_size: int) -> int:
    n = blocks_for(max_prompt + max_generated, block_size) + 1
    return n + (n % 2)

assert blocks_from_hint(30_000, 2_000, BS) == 126     # 32k-token request: 126 blocks, not 32
assert blocks_from_hint(4_000, 500, BS) == 20         # ceil(4500/256)=18, +1=19, rounded to 20

print("paged block allocator OK")
print(f"  cache rows = (num_blocks + {RESERVED_BLOCKS}) * block_size = (64 + 2) * {BS} = {cache.shape[0]}")
print(f"  read_trash={read_trash} sentinel={sentinel} write_trash={write_trash} (never allocated)")
print(f"  internal fragmentation <= block_size - 1 = {BS - 1} tokens per sequence")
print()
print("  block_size | num_blocks | reserved % of pool | block-table rows (8k seq) | worst tail waste")
for bs, nb, rf, te, fr in rows:
    print(f"  {bs:>10} | {nb:>10} | {rf * 100:>17.4f}% | {te:>25} | {fr:>16}")

Executed output:

paged block allocator OK
  cache rows = (num_blocks + 2) * block_size = (64 + 2) * 256 = 16896
  read_trash=16384 sentinel=16385 write_trash=16640 (never allocated)
  internal fragmentation <= block_size - 1 = 255 tokens per sequence

  block_size | num_blocks | reserved % of pool | block-table rows (8k seq) | worst tail waste
           4 |     262142 |            0.0008% |                      2048 |                3
          16 |      65534 |            0.0031% |                       512 |               15
          64 |      16382 |            0.0122% |                       128 |               63
         256 |       4094 |            0.0488% |                        32 |              255
        1024 |       1022 |            0.1953% |                         8 |             1023

Two operational facts fall out. Internal fragmentation is at most block_size - 1 tokens per sequence, so at the default 256 a workload of many short prompts pays up to 255 wasted token slots each; that is the case where a smaller block earns its keep. And the default block table addresses exactly 32 x 256 = 8192 tokens: a request longer than 8192 tokens needs either a workload hint (max_prompt_length plus max_generated_length) or an explicit max_blocks_per_request, or its block table cannot address its own KV.

Keep block_size at 256 unless measurement says otherwise. FlashAttention's paged KV-cache path requires a page block size that is a multiple of 256; the default is the smallest compatible value, not the only valid value.

How to use it¶

The blocking path. This is a reference template: it needs a GPU and transformers==5.13.1, and was not executed for this page.

# Reference template (needs GPU + transformers==5.13.1). Not executed here.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from transformers.generation import ContinuousBatchingConfig, GenerationConfig

model = AutoModelForCausalLM.from_pretrained(
    "Qwen/Qwen3-4B",
    attn_implementation="paged|flash_attention_2",   # paged| is added automatically if omitted
    device_map="cuda",
    dtype=torch.bfloat16,
)
tokenizer = AutoTokenizer.from_pretrained("Qwen/Qwen3-4B")

inputs = [tokenizer.encode(p) for p in ["Whats up?", "Name a cat breed."]]

cb_config = ContinuousBatchingConfig(
    block_size=256,          # KV block size in tokens; must be >= 4
    max_batch_tokens=8192,   # per-forward-pass query-token budget
    scheduler_type="fifo",   # or "prefill_first"
    safety_margin=0.15,      # admission reserve; 0.15 is the fifo default
)
generation_config = GenerationConfig(max_new_tokens=64, eos_token_id=tokenizer.eos_token_id)

outputs = model.generate_batch(
    inputs=inputs,
    generation_config=generation_config,
    continuous_batching_config=cb_config,
)
for request_id, output in outputs.items():
    print(request_id, tokenizer.decode(output.generated_tokens, skip_special_tokens=True))

The serving path is an OpenAI-compatible HTTP endpoint built on the same manager:

# Reference template. transformers serve, continuous batching on the SDPA paged backend.
transformers serve \
  --continuous-batching \
  --attn-implementation "sdpa"

# flash_attention_2 instead of sdpa is what unlocks the decode fast path on CUDA.
transformers serve \
  --continuous-batching \
  --attn-implementation "flash_attention_2"

The config that matters¶

ContinuousBatchingConfig defaults, read off main on 2026-07-13. Most are sentinels resolved at startup from free VRAM and workload hints, so the effective value is not the literal default.

Field	Default	Resolves to
`block_size`	`256`	Literal. Below 4 raises `ValueError`
`num_blocks`	`None`	Solved from free VRAM after weights load
`max_batch_tokens`	`None`	8192, clamped by VRAM, floor 256
`max_memory_percent`	`None`	0.9, or 0.8 when logits processors are active
`max_requests_per_batch`	`None`	Number of prompts submitted; fallback 1024; then capped by `max_batch_tokens` and `num_blocks`
`max_blocks_per_request`	`None`	From workload hints, else 32. `0` disables the decode fast path, and it is forced to `0` on any sliding-window model
`allow_block_sharing`	`True`	Prefix caching. Forced off unless every layer is full-attention
`use_async_batching`	`None`	On iff CUDA graphs are on and no attention mask is needed
`use_cuda_graph`	`None`	On iff no attention mask is needed (i.e. FlashAttention)
`default_compile_level`	`0`	0 none, 1 `default`+dynamic, 2 `max-autotune-no-cudagraphs`+dynamic, 3 same, `dynamic=False`
`scheduler_type`	`"fifo"`	`SCHEDULER_MAPPING` = `{"fifo": FIFOScheduler, "prefill_first": PrefillFirstScheduler}`
`safety_margin`	`None`	0.15 under `fifo`, 0.0 under `prefill_first`
`cpu_offload_space`	`0.0`	GiB of pinned CPU swap. 0.0 disables the CPU tier
`cpu_offload_space_safety_threshold`	`0.8`	Clamps the swap pool to 80% of available RAM (needs `psutil`)
`return_logprobs`	`False`	Per-token logprobs in the output
`per_request_processors`	`False`	Per-request `temperature` / `top_k` / `top_p`
`max_queue_size`	`0`	Unlimited

use_default_compile_configs (deprecated in 5.11) and max_cached_graphs (deprecated in 5.13) still exist and log warnings.

How to develop with it¶

The manager is the API you build a server on. Requests are submitted and collected independently of the generation loop.

# Reference template (needs GPU + transformers==5.13.1). Not executed here.
from transformers.generation.continuous_batching import RequestStatus

# Scoped: started on enter, stopped (and destroyed) on exit.
with model.continuous_batching_context_manager(generation_config=generation_config) as manager:
    manager.add_request(input_ids=tokenizer.encode("Write a history of QM."),
                        request_id="long", max_new_tokens=512)
    manager.add_request(input_ids=tokenizer.encode("What's up?"),
                        request_id="short", max_new_tokens=32, streaming=True)

    # Token-by-token for one request.
    for chunk in manager.request_id_iter(request_id="short"):
        print(tokenizer.decode(chunk.generated_tokens[-1:]), end="", flush=True)
        if chunk.status == RequestStatus.FINISHED:
            break

    # Or drain everything as it completes.
    for result in manager:
        print(result.request_id, tokenizer.decode(result.generated_tokens))

Four points that are easy to get wrong:

add_requests sorts its inputs to maximise prefix-cache hits when block sharing is on. Submitting one at a time with add_request forfeits that.
get_result(request_id=...) is not a lookup. It pops the next result from the output queue; if it does not match the requested id, the result is re-queued and the call returns None. Iterate, or register a handler with register_result_handler, rather than polling a specific id in a loop.
stop() is graceful, stop(hard_stop=True) is not. The graceful path drains queued and active requests; the hard path fails them with a RuntimeError. After stop() the manager can be restarted with start(). destroy() releases distributed resources and is terminal.
Per-request sampling needs per_request_processors=True, and the corresponding GenerationConfig field must be non-default (for example temperature set to something other than None or 1), or the processor is never built and the per-request value is silently ignored.

For RL rollouts, return_logprobs=True returns the log-softmax of the processed logits alongside the tokens, which is what removes the need to re-score rollouts on a second engine. See rollout redundancy for the surrounding cost model.

How to maintain it¶

Pin transformers exactly. This subsystem is moving fast: block_size was 32 before it became 256, max_cached_graphs was deprecated in 5.13, use_default_compile_configs in 5.11. Treat a minor bump as a behaviour change and re-read ContinuousBatchingConfig.
Assert the decode fast path is live, do not assume it. max_blocks_per_request is silently set to 0 by two independent code paths. First, ensure_decode_fast_path_is_available zeroes it whenever the attention backend cannot provide flash_attn_with_kvcache, and logs a warning only if you set max_blocks_per_request explicitly, so setting it explicitly is the cheapest way to make that failure loud. Second, and later, continuous_api.py zeroes it unconditionally when the cache reports num_sliding_attention_groups > 0. That one runs after the availability check, so it fires no warning at all, not even the explicit-set one. Read the resolved config back after startup instead of relying on the log.
Re-check allow_block_sharing after a model swap. Prefix caching is force-disabled for any model with a sliding-window layer group (use_prefix_sharing = allow_block_sharing and group_types == ["full_attention"]). A model change from full-attention to hybrid attention therefore costs both prefix reuse and the decode fast path, and neither loss is announced.
Size the block table for the longest real request. The 32-block fallback covers 8192 tokens. Pass workload hints or set max_blocks_per_request when requests are longer.
Watch the prefix cache under memory pressure. Cached complete blocks count as free: when uninitialized blocks run out, the manager lazily demotes the most recently initialized block back to uninitialized and drops its hash. This is a best-effort cache, not a reservation, and its eviction order is the opposite of vLLM's LRU (KV cache management).

How to run it in production¶

Schedulers and the admission reserve¶

Two schedulers ship. FIFOScheduler (default) fills the batch in arrival order with active decodes first, then active prefills, then new waiting requests, which prioritises finishing in-flight work. PrefillFirstScheduler completes split (chunked) prefills before resuming decode, which reduces fragmentation on long-prompt workloads.

The knob that has no direct equivalent in vLLM or SGLang is safety_margin. Once free blocks fall below safety_margin * num_blocks, the scheduler holds additional prefills while continuing active work. A liveness exception admits the first prefill when the current step would otherwise schedule no request. That is an explicit admission reserve (inference QoS and admission control), and it defaults to 0.15 under fifo and 0.0 under prefill_first.

It is worth understanding exactly what the reserve does and does not buy, because it is a reserve against transients, not a capacity substitute. The block below models the pool under two workloads.

# Runnable on system python3 (numpy). safety_margin is an ADMISSION RESERVE: when
# free_blocks < safety_margin * num_blocks this simplified model holds new PREFILLS but
# keeps DECODING what is already in flight. The production scheduler also admits one prefill
# when a step would otherwise be empty, a liveness exception not exercised by this workload. Without it, admissions drive the pool to zero and
# a decode that needs its next block is evicted (offloaded to CPU, or soft reset: generated
# tokens are appended to the prompt, blocks are freed, the request is re-queued).
import numpy as np

BLOCK_SIZE = 256
NUM_BLOCKS = 64


def blocks_for(tokens: int) -> int:
    return (tokens + BLOCK_SIZE - 1) // BLOCK_SIZE


class Pool:
    """Simplified block pool for the admission-reserve mechanism."""

    def __init__(self, num_blocks: int, safety_margin: float):
        if not 0.0 <= safety_margin <= 1.0:
            raise ValueError(f"Got safety_margin = {safety_margin} but expected a value in [0, 1]")
        self.num_blocks = num_blocks
        self.free = num_blocks
        self.margin = safety_margin
        self.evictions = 0
        self.admitted = 0
        self.first_eviction: int | None = None

    @property
    def reserve(self) -> int:
        """Smallest free-block count that still admits a prefill: ceil(margin * num_blocks)."""
        return int(np.ceil(self.margin * self.num_blocks))

    def may_admit(self, need: int) -> bool:
        """Prefills are held back below the reserve. Decodes are not: see the loop below."""
        return self.free >= need and self.free >= self.margin * self.num_blocks


def simulate(margin: float, arrival_every: int, steps: int, prompt: int, gen: int) -> Pool:
    """Each request prefills `prompt` tokens, decodes `gen` more, takes another block every
    time it crosses a block boundary, and frees everything when it finishes. Deterministic."""
    pool = Pool(NUM_BLOCKS, margin)
    live: list[list[int]] = []                          # [tokens, blocks_held]
    for step in range(steps):
        for req in live:                                # 1. decodes grow first (FIFO priority)
            req[0] += 1
            if blocks_for(req[0]) > req[1]:             # crossed a boundary: one more block
                if pool.free == 0:                      # pool empty: EVICT this request
                    pool.evictions += 1
                    if pool.first_eviction is None:
                        pool.first_eviction = step
                    pool.free += req[1]
                    req[0], req[1] = -1, 0              # re-queued, its KV is gone
                    continue
                pool.free -= 1
                req[1] += 1
            elif req[0] >= prompt + gen:                # 2. finished: return its blocks
                pool.free += req[1]
                req[0], req[1] = -1, 0
        live = [r for r in live if r[0] >= 0]
        need = blocks_for(prompt)                       # 3. only then admit a new prefill
        if step % arrival_every == 0 and pool.may_admit(need):
            pool.free -= need
            pool.admitted += 1
            live.append([prompt, need])
    return pool


# Growth demand: blocks a request still needs AFTER admission, i.e. what the reserve must cover.
def growth_blocks(prompt: int, gen: int) -> int:
    return blocks_for(prompt + gen) - blocks_for(prompt)


SHORT = dict(arrival_every=5, steps=600, prompt=200, gen=200)   # 1 block of growth per request
LONG = dict(arrival_every=5, steps=600, prompt=200, gen=800)    # 3 blocks of growth per request

# 1. Short generations: the reserve prevents eviction outright. Zero margin does not.
s0, s15 = simulate(0.00, **SHORT), simulate(0.15, **SHORT)
assert (s0.admitted, s0.evictions) == (120, 9), (s0.admitted, s0.evictions)
assert (s15.admitted, s15.evictions) == (96, 0), (s15.admitted, s15.evictions)
assert s15.first_eviction is None

# 2. It is not free: the reserve gives up 24 admissions to buy those 9 avoided evictions.
assert s0.admitted - s15.admitted == 24

# 3. Long generations: the reserve REDUCES and DELAYS eviction but does not abolish it.
#    A margin is a buffer against transients, not a substitute for capacity.
l0, l15 = simulate(0.00, **LONG), simulate(0.15, **LONG)
assert (l0.evictions, l15.evictions) == (59, 13), (l0.evictions, l15.evictions)
assert l15.evictions < l0.evictions
assert l15.first_eviction > l0.first_eviction            # eviction arrives later, still arrives

# 4. WHY, quantified. The reserve must cover the growth still owed to in-flight decodes.
#    reserve = ceil(0.15 * 64) = 10 blocks. A short request owes 1 more block, so the reserve
#    covers 10 of them concurrently. A long request owes 3, so it covers only 3, and the sim
#    keeps far more than 3 in flight: the reserve is structurally too small for that workload.
reserve = Pool(NUM_BLOCKS, 0.15).reserve
assert reserve == 10                                     # ceil(0.15 * 64) = ceil(9.6)
assert growth_blocks(200, 200) == 1 and growth_blocks(200, 800) == 3
assert reserve // growth_blocks(200, 200) == 10          # 10 concurrent short decodes covered
assert reserve // growth_blocks(200, 800) == 3           # only 3 concurrent long decodes covered

# 5. The reserve gates PREFILLS only. One block under the margin, a new prefill is refused
#    while an in-flight decode still gets its block. That asymmetry is the whole mechanism.
p = Pool(NUM_BLOCKS, 0.15)
p.free = reserve
assert p.may_admit(1) is True                            # exactly at the reserve: admissible
p.free -= 1
assert p.may_admit(1) is False                           # below it: prefill held back
assert p.free > 0                                        # ...but blocks remain for decodes

# 6. Boundary: margin = 1.0 reserves the whole pool, so a prefill is admitted ONLY into a
#    completely idle pool. Concurrency collapses to one request. A total reserve is
#    serialization, not safety.
full = simulate(1.00, **SHORT)
assert full.evictions == 0 and full.admitted <= 3        # 600 steps, one request at a time
q = Pool(NUM_BLOCKS, 1.0)
assert q.may_admit(1) is True                            # pool completely free
q.free -= 1
assert q.may_admit(1) is False                           # a single block in use blocks admission

# 7. Adversarial: out-of-range margins must raise, matching Scheduler.__init__.
for bad in (-0.01, 1.01, 2.0):
    raised = False
    try:
        Pool(NUM_BLOCKS, bad)
    except ValueError:
        raised = True
    assert raised, f"safety_margin={bad} must raise"

print("safety_margin admission reserve OK")
print(f"  pool = {NUM_BLOCKS} blocks x {BLOCK_SIZE} tokens, reserve at 0.15 = {reserve} blocks")
print()
print("  workload                       | margin | admitted | evictions | 1st eviction")
for tag, cfg, pl in (("short gen (owes 1 block) ", SHORT, s0), ("short gen (owes 1 block) ", SHORT, s15),
                     ("long gen  (owes 3 blocks)", LONG, l0), ("long gen  (owes 3 blocks)", LONG, l15)):
    ev = pl.first_eviction if pl.first_eviction is not None else "never"
    print(f"  {tag}      | {pl.margin:>6.2f} | {pl.admitted:>8} | {pl.evictions:>9} | {ev:>12}")
print()
print(f"  short: reserve ({reserve}) covers {reserve // 1} concurrent decodes -> 0 evictions")
print(f"  long:  reserve ({reserve}) covers only {reserve // 3} -> eviction is delayed"
      f" ({l0.first_eviction} -> {l15.first_eviction}), not avoided ({l0.evictions} -> {l15.evictions})")

Executed output:

safety_margin admission reserve OK
  pool = 64 blocks x 256 tokens, reserve at 0.15 = 10 blocks

  workload                       | margin | admitted | evictions | 1st eviction
  short gen (owes 1 block)       |   0.00 |      120 |         9 |          187
  short gen (owes 1 block)       |   0.15 |       96 |         0 |        never
  long gen  (owes 3 blocks)      |   0.00 |       83 |        59 |          187
  long gen  (owes 3 blocks)      |   0.15 |       33 |        13 |          212

  short: reserve (10) covers 10 concurrent decodes -> 0 evictions
  long:  reserve (10) covers only 3 -> eviction is delayed (187 -> 212), not avoided (59 -> 13)

The operating rule: the reserve only holds while it covers the block growth still owed to the requests already in flight. At 64 blocks and a 0.15 margin the reserve is 10 blocks, which covers ten short decodes but only three long ones. On long-generation workloads a bigger safety_margin buys a later and smaller eviction storm, not a cure; that needs capacity, a CPU offload tier, or fewer concurrent requests.

The response to KV pressure, and how it differs from vLLM¶

When an in-flight decode cannot allocate its next block, transformers does not immediately drop work. The offloading manager takes victims from the requests the scheduler reports as starved, newest first (so the batch just scheduled keeps its cache), never evicting the last remaining active request, and it evicts only as many as the demand requires. Each victim then takes one of two paths:

CPU offload. If cpu_offload_space > 0.0 and the victim fits in the pinned pool, its KV blocks are copied to host RAM in a single batched transfer and copied back when GPU space frees up. The request resumes without recomputing its prompt or its generated tokens.
Soft reset. Otherwise, the request's generated tokens are appended to its prompt, its blocks are freed, and it is re-queued with the same request_id. On resume it re-prefills its (now longer) prompt. This is recomputation.

This differs from vLLM only when CPU offload is enabled: vLLM V1 uses RECOMPUTE, and its CPU-swap path was removed as dead code (KV cache management, continuous batching internals). The transformers default is cpu_offload_space=0.0, so its pressure response also uses recomputation, although the engines' scheduling and victim-selection policies still differ. The swap tier is opt-in, and it is the thing to turn on when soft resets are costing measurable prefill.

Async batching: the one published benchmark, read carefully¶

By default the loop is synchronous: the CPU prepares a batch, the GPU computes it, the CPU reads the results back, and neither ever works while the other does. Async batching (use_async_batching) overlaps them using three CUDA streams and three events, so the CPU can prepare batch N+1 while the GPU computes batch N. The source symbols are h2d_stream, d2h_stream, compute_stream and h2d_over, compute_over, d2h_over, all in continuous_batching/input_outputs.py; the blog calls the events h2d_done / compute_done / d2h_done, which is a name that appears nowhere in the code, so grep for the _over forms. Two mechanisms make it safe:

Dual buffer slots (A/B). The CPU cannot write batch N+1's inputs into buffers the GPU is still reading, so the engine alternates between two sets of I/O tensors. This doubles the RAM and VRAM for input and output tensors, which is the real cost of the feature. It is cheapest under FlashAttention, which needs no attention mask (by far the largest input tensor).
Two CUDA graphs, one memory pool. A graph is captured against fixed addresses, so slot A's graph cannot replay on slot B's buffers. Both graphs allocate from a shared pool, so total VRAM is capped at the max across graphs rather than their sum. The constraint (two graphs in one pool must never run concurrently) is satisfied because batch N finishes before N+1 starts.
Carry-over. Batch N+1's inputs are built before batch N has produced its tokens, so the missing token is a placeholder 0, and a carry-over mask (holding a destination index, or -1 for "do not carry") adds the real token in before the forward pass. The four carry-over ops are cheap enough to be captured inside the CUDA graph.

Async batching requires CUDA graphs, because graph replay is what provides the stable tensor addresses that stream overlap needs.

The published result, and every caveat that belongs with it:

Metric	Sync	Async
Total generation time	300.6 s	234.5 s
GPU active fraction of wall clock	76.0%	99.4%

Conditions, quoted in full because they are all that is given: "8K tokens, batch size 32, 8B model". Read this as follows:

Hardware is never stated and the model is never named. The H200 in the post is an unrelated pricing aside, not the benchmark machine. Any table reproducing this must carry "hardware not disclosed".
"99.4% GPU utilization" is GPU busy-time from a span trace. It is not SM occupancy and not MFU. Call it the GPU active fraction of wall clock and nothing stronger; it says the GPU had work queued, not that the work was efficient.
The "22% speedup" is a time reduction, not a throughput gain. 300.6 s to 234.5 s is a 22.0% reduction in time, which is a 28.2% throughput increase (1.28x). Quoting the 22% next to another engine's tokens-per-second figure compares two different quantities.
No repeat count or variance is disclosed. The post reports one synchronous result and one asynchronous result, with no tokens per second, TTFT, inter-token latency, percentiles, or comparison against another engine.

The mechanism generalises even though the number does not: the win exists only while the CPU finishes preparing batch N+1 before the GPU finishes computing batch N. On a small model, a fast GPU, or a batch small enough that the forward pass is short, the CPU becomes the critical path again and the overlap buys nothing while still costing double the I/O memory. See CUDA streams and concurrency.

Failure modes¶

Symptom	Cause	Action
Decode much slower than expected, no error or warning	The decode fast path was silently disabled: `flash_attn_with_kvcache` is unavailable for this backend/device, so `ensure_decode_fast_path_is_available` set `max_blocks_per_request` to `0` and every batch takes the varlen path	Set `max_blocks_per_request` explicitly, which turns the silent fallback into a logged warning. The fast path needs FA2/FA3 on CUDA, or FA2 on XPU
Decode slow on a sliding-window model, and setting `max_blocks_per_request` explicitly produces no warning either	A second, independent kill switch: after the cache is built, `continuous_api.py` zeroes `max_blocks_per_request` outright when `num_sliding_attention_groups > 0` (a TODO in the source). It runs after `ensure_decode_fast_path_is_available`, so the explicit-set warning never fires	Assert on the resolved config after startup rather than trusting a warning. A sliding-window model loses the fast path even on FA2/CUDA; there is no config that turns it back on
No prefix-cache reuse on a model with a shared system prompt	The model has a sliding-window layer group, so `use_prefix_sharing` is forced off regardless of `allow_block_sharing`	Expect no reuse; do not spend memory chasing it. A sliding-window model loses the decode fast path as well (row above), so it is two losses, not one. Route to an engine with a different cache design if either matters
`ValueError: Block size must be at least 4`	`block_size` below the implementation minimum	Use 256 (the default) unless a measurement justifies otherwise
Requests longer than 8192 tokens misbehave on the fast path	The default block table is 32 blocks x 256 tokens = 8192	Pass workload hints (`max_prompt_length`, `max_generated_length`) or set `max_blocks_per_request`
Throughput collapses under load, prefill time balloons	Victims are being soft reset (recomputed) because `cpu_offload_space=0.0`	Raise capacity, lower concurrency, raise `safety_margin`, or enable the CPU swap tier
A bigger `safety_margin` does not stop eviction	The reserve does not cover the block growth still owed to in-flight decodes (validated above)	Long generations need capacity or offload, not a bigger reserve
OOM shortly after enabling async batching	Dual I/O slots double the input/output tensor memory	Disable `use_async_batching`, or lower `max_batch_tokens` / `max_requests_per_batch`
OOM during sampling on a large-vocabulary model	The logits tensor scales with batch tokens x vocab and is cast to fp32	Lower `max_requests_per_batch`; note `max_memory_percent` already drops 0.9 to 0.8 when logits processors are active
Per-request `temperature` silently ignored	`per_request_processors=False`, or the `GenerationConfig` field is at its default so the processor was never built	Set `per_request_processors=True` and a non-default value in `GenerationConfig`
`get_result(request_id=...)` returns `None` for a live request	It pops the queue head and re-queues on mismatch; it is not a lookup	Iterate the manager, use `request_id_iter`, or `register_result_handler`
Tensor-parallel startup error about KV heads	TP size must divide `num_key_value_heads`	Choose `--nproc-per-node` accordingly

Open questions and validation¶

Compile and continuous batching. The transformers serve docs state flatly that "Compile is incompatible with continuous batching", while ContinuousBatchingConfig exposes default_compile_level 1 to 3 with separate varlen_compile_config and decode_compile_config, and resolve_compile_configs clearly builds them. These cannot both be complete descriptions. The likely reading is that the serve CLI's --compile flag drives a different code path from the config's compile levels, but that is inference, not a verified fact. Do not rely on either statement; test compile on your own model and measure.
Upstream doc drift, verified 2026-07-13. The async blog says "the general entry point for continuous batching is continuous_batching.py"; no such file exists (it is the package continuous_batching/, with continuous_api.py as the entry point). The continuous_api.py module docstring points at ContinuousBatchingConfig.resolve_sentinel_values(), a method that does not exist (defaults resolve in initialization.py). Both continuous_batching.md and continuous_batching_architecture.md claim the first blog post covers "performance benchmark numbers"; it contains none. continuous_batching_architecture.md says async batching "uses two I/O buffer pairs and two CUDA streams"; the code creates three (h2d_stream, d2h_stream, compute_stream, input_outputs.py), and the async blog agrees it is three, so the architecture doc is the stale one. The blog's event names (h2d_done, compute_done, d2h_done) are also not the code's (h2d_over, compute_over, d2h_over).
Errors in the first blog post, not propagated here. It calls the (n-1)(B-1) padding cost "quadratic" (it is linear in each variable, refuted in the executed block above); its KV example ("16 KB in memory") is per token per layer, so the per-token total for Llama-2-7B across 32 layers is 512 KB, and the "16 KB per token" reading is wrong; and it motivates chunked prefill as a memory constraint, when the production reason is head-of-line blocking and decode starvation (Sarathi-Serve), which continuous batching internals already states correctly.
Prefix-cache eviction order is LIFO, not LRU. The manager demotes the most recently initialized block first. This is what the architecture doc says and what BlockManager does, but the consequence (a hot, long-lived shared prefix is evicted later than a recently added one, which is desirable, while recently built prefixes are the first to go) is not discussed upstream and is not benchmarked. Measure hit rate before relying on it.
No transformers-versus-vLLM throughput number is citable. None is published under a reproducible method. Any capacity decision between the two engines needs a local A/B on real traffic.

References¶

Hugging Face, Continuous batching (docs): generate_batch, ContinuousBatchingManager, ContinuousBatchingConfig fields and defaults, the paged-backend table, tensor parallelism, sliding-window behaviour.
Hugging Face, Continuous batching architecture (docs): request lifecycle, chunked prefill, FIFO and PrefillFirst schedulers, admission and safety_margin, prefix caching, lazy eviction, soft reset, CUDA graphs, async batching, offloading.
Hugging Face, Paged attention (docs): the varlen path (flash_attn_varlen_func) and the decode fast path (flash_attn_with_kvcache), block tables, cache_seqlens.
Hugging Face, transformers source: generation/continuous_batching/: cache.py (_min_block_size = 4, the num_blocks + 2 reserved zone, _default_max_batch_tokens = 8192, _min_max_batch_tokens = 256), scheduler.py (SCHEDULER_MAPPING, safety-margin defaults 0.15 / 0.0), initialization.py (FALLBACK_DEFAULTS, decode-fast-path availability, CUDA-graph and async auto-detection), offloading_manager.py (newest-first, demand-driven offload), cache_manager.py (content-hashed, ref-counted, lazily evicted blocks).
Hugging Face, Server optimizations (transformers serve): --continuous-batching, --attn-implementation, and the "Compile is incompatible with continuous batching" note.
Rémi Ouazan Reboul, Arthur Zucker, Luc Georges, Continuous batching from first principles, Hugging Face, 2025-11-25. Pedagogical only: no API, no code, no benchmark numbers. Source of the (n-1)(B-1) derivation and the 693-token example, and of the three errors listed under Open questions.
Rémi Ouazan Reboul, Pedro Cuenca, Aritra Roy Gosthipaty, Unlocking asynchronicity in continuous batching, Hugging Face, 2026-05-14. Source of the CUDA-streams design and the reported comparison (300.6 s to 234.5 s; 76.0% to 99.4% GPU active fraction; "8K tokens, batch size 32, 8B model"; hardware, model name, repeat count, and variance not disclosed).
Yu, Jeong, Kim, Kim, Chun, Orca: A Distributed Serving System for Transformer-Based Generative Models, OSDI '22: iteration-level scheduling and selective batching, the origin of continuous batching.