Markdown

Cookbook: ThunderAgent agentic serving in front of vLLM¶

Scope: the concrete deployment recipe for program-aware agentic inference, from a running vLLM backend to an agent client whose multi-turn calls the scheduler can keep resident. This page is the operational how-to: the commands to launch the middleware, the three client-side changes, an executed KV-cache capacity planner that tells you how many concurrent programs a backend holds before it thrashes, and the wiring for an RL rollout. The concepts, the pause/restore scheduler, and the eviction proof live on the tech page; the rollout-workload framing is agentic RL and async RL systems.

The vllm, thunderagent, and client snippets are reference templates on the real interfaces (they need the installed packages and a GPU backend, and flags move between versions, so verify against the current repo). The Python capacity planner is executed and asserted (stdlib only); its per-model layer and head counts are illustrative placeholders you must replace with your model's real config.

flowchart LR
  subgraph SETUP["Deploy once"]
    S1["1. vLLM serves the model<br/>:8000"] --> S2["2. thunderagent fronts it<br/>:9000"]
  end
  S2 --> S3["3. Client sends to :9000<br/>with program_id"]
  S3 --> S4["4. Size concurrency<br/>KV capacity planner"]
  S4 --> S5["5. Run rollout<br/>(SkyRL / slime)"]
  S5 --> S6["6. Watch metrics<br/>hit rate, imbalance, disk"]
  S6 -.->|"tune dt, decay,<br/>node count"| S2

What it is¶

A recipe for inserting ThunderAgent as a scheduling and tool-management layer between your agent clients and one or more inference backends, without replacing the backend. You keep vLLM (or SGLang) exactly as you run it today; ThunderAgent listens on a new port, forwards requests to the backend, and uses a program_id on each call to schedule whole workflows instead of isolated requests. The payoff, the mechanics, and the reported speedups are on the tech page; this page is the sequence of steps and the sizing math.

Why use it¶

You already have a vLLM or SGLang deployment and want the program-aware wins (fewer KV re-prefills, balanced memory across nodes, reclaimed tool sandboxes) without rewriting the serving stack. The integration is three client-side changes and one extra process.
Your rollouts thrash at high concurrency. When many long agentic trajectories run at once, their combined KV cache exceeds a backend's capacity and a request scheduler evicts and re-prefills; sizing concurrency against the planner below tells you when that happens and how pausing avoids it.
Your tool sandboxes are heavy or leaky. The tool manager reclaims sandboxes on workflow completion and prepares them asynchronously, which matters most when a sandbox is gigabytes and slow to start.

When to use it (and when not)¶

Use this recipe for multi-turn agentic serving or agentic RL rollout at concurrency high enough that resident contexts contend for backend memory. Skip it for single-turn or stateless serving (nothing persists across turns to schedule), and re-measure before trusting a speedup on stochastic-tool workloads, where the tech page documents sub-parity points. If you are not yet contending for KV capacity, the capacity planner below will tell you: if your arriving program count is well under fits, plain vLLM with prefix caching is enough.

Architecture¶

Three planes stay separate. The backend plane is the unchanged vLLM or SGLang engine holding model weights and the KV cache. The scheduling plane is ThunderAgent: it terminates the OpenAI-style client connection, attributes each call to a program, and issues pause/restore against the backend. The tool plane is the sandbox fleet (Docker containers on a CPU cluster in the paper's deployment) that the resource manager prepares and reclaims. The client talks only to the scheduling plane's port.

How to use it¶

1. Serve the model with a backend engine¶

Bring up vLLM as you normally would; ThunderAgent does not change how the model is loaded or quantized.

# Reference template. Serve the policy model on the backend port.
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen3-32B --port 8000 \
  --tensor-parallel-size 8            # match your GPU count / model size

2. Front it with ThunderAgent¶

Install ThunderAgent and point it at the backend. Clients will send to port 9000 from here on.

# Reference template (repo README).
git clone https://github.com/ThunderAgent-org/ThunderAgent.git
cd ThunderAgent && pip install -e .
thunderagent --backend-type vllm --backends http://localhost:8000 \
             --port 9000 --metrics --profile
# --metrics/--profile expose the per-program token, tool-time, and hit-rate views.

For a multi-node backend, pass several --backends URLs so the global program-aware waiting queue can balance restores across replicas.

3. Tag client calls with program_id (the three changes)¶

The only client edits: set base_url to the ThunderAgent port, add program_id (and optional docker_ids) to each call, and release the program when the workflow ends so the GC reclaims its sandbox.

# Reference template (repo README + paper Appendix B). Three changes total.
import openai
client = openai.OpenAI(base_url="http://localhost:9000/v1", api_key="x")  # (1) port

def agent_turn(messages, program_id, sandbox_ids):
    extra_body = {"program_id": program_id}          # (2) attribute to a workflow
    if sandbox_ids:
        extra_body["docker_ids"] = sandbox_ids        #     hand over tool assets
    return client.chat.completions.create(
        model="Qwen/Qwen3-32B", messages=messages, extra_body=extra_body,
    )

# (3) On completion: POST /programs/release {"program_id": program_id}
#     so the hook-based GC tears down the sandbox and frees disk.

4. Size concurrency with the KV-cache capacity planner¶

Before you launch a large rollout, work out how many programs a backend can hold resident before it thrashes. That number, fits, is the concurrency at which a request-level scheduler starts evicting and re-prefilling; ThunderAgent instead pauses the excess. This planner is executed and self-checking; replace the illustrative model config with your model's real layer and head counts:

# thunder_capacity.py -- validated: size how many concurrent agentic programs a
# backend can hold before its KV cache thrashes, and show what program-aware
# pausing buys over a request-level scheduler that just OOMs. Pure stdlib.
#
# The KV-cache formula is standard and verifiable; the specific layer/head numbers
# below are ILLUSTRATIVE placeholders -- plug your own model's config.

GiB = 1024 ** 3


def kv_bytes_per_token(n_layers: int, n_kv_heads: int, head_dim: int, dtype_bytes: int) -> int:
    """Resident KV cache per context token: K and V (hence 2x), for every layer,
    across the key/value attention heads. This is the standard accounting an
    inference engine uses to budget its cache."""
    return 2 * n_layers * n_kv_heads * head_dim * dtype_bytes


def kv_token_budget(hbm_bytes: int, weight_bytes: int, reserve_frac: float = 0.10) -> float:
    """Tokens of KV cache a backend can hold: HBM minus weights minus a reserve
    for activations/fragmentation, divided by bytes-per-token happens in caller."""
    free = hbm_bytes * (1.0 - reserve_frac) - weight_bytes
    return max(0.0, free)


def max_resident_programs(free_kv_bytes: float, bytes_per_tok: int, avg_context: int) -> int:
    """How many programs of avg_context tokens fit resident at once."""
    if avg_context <= 0:
        return 0
    return int(free_kv_bytes // (bytes_per_tok * avg_context))


# --- Illustrative model + hardware (replace with your model's real config) ------
LAYERS, KV_HEADS, HEAD_DIM = 64, 8, 128     # a GQA-style ~30B-class placeholder
BF16 = 2
bpt = kv_bytes_per_token(LAYERS, KV_HEADS, HEAD_DIM, BF16)
assert bpt == 2 * 64 * 8 * 128 * 2 == 262_144   # 256 KiB / token

HBM = 80 * GiB          # one H100-80GB
WEIGHTS = 32 * GiB      # ~32 GiB of FP8/BF16 weights on this device (illustrative)
free = kv_token_budget(HBM, WEIGHTS)
resident_tokens = free / bpt
AVG_CTX = 8000          # a mid-rollout agentic context

fits = max_resident_programs(free, bpt, AVG_CTX)
# ~40 GiB of KV budget / 256 KiB per token ~ 163k resident tokens ~ 20 programs.
assert 18 <= fits <= 22, fits

# --- Sanity cross-checks on the formula -----------------------------------------
# bf16 KV is exactly half of fp32 KV (dtype linearity).
assert kv_bytes_per_token(LAYERS, KV_HEADS, HEAD_DIM, 4) == 2 * bpt
# Monotonic: more HBM never reduces the resident program count.
assert max_resident_programs(kv_token_budget(96 * GiB, WEIGHTS), bpt, AVG_CTX) >= fits
# Boundary: weights that fill HBM leave no KV budget and zero programs, no crash.
assert max_resident_programs(kv_token_budget(HBM, HBM), bpt, AVG_CTX) == 0
# Longer contexts hold fewer programs (KV grows linearly with context length).
assert max_resident_programs(free, bpt, 2 * AVG_CTX) <= fits // 2 + 1

# --- Program-aware pausing vs request-level OOM ---------------------------------
# A request-level scheduler admits all N arriving programs; once resident tokens
# exceed the budget it thrashes (evict + reprefill) or OOMs. ThunderAgent instead
# keeps at most `fits` programs Active and Pauses the rest, so N programs make
# progress on a backend that can only hold `fits` of them resident at once.
def active_and_paused(arriving: int, capacity_programs: int) -> tuple[int, int]:
    active = min(arriving, capacity_programs)
    return active, max(0, arriving - active)

N = 50
active, paused = active_and_paused(N, fits)
assert active == fits and paused == N - fits          # e.g. 20 active, 30 paused
# The paused programs' KV is evicted, so the resident token count stays within
# budget instead of overflowing it -- this is what avoids the thrash.
assert active * AVG_CTX <= resident_tokens

print(f"OK: {bpt // 1024} KiB/token; {free / GiB:.1f} GiB KV budget holds "
      f"~{resident_tokens / 1000:.0f}k tokens = {fits} programs of {AVG_CTX} tok resident. "
      f"For {N} arriving programs, program-aware keeps {active} Active + {paused} Paused "
      f"within budget; request-level would overrun it and thrash.")

Run output:

OK: 256 KiB/token; 40.0 GiB KV budget holds ~164k tokens = 20 programs of 8000 tok resident. For 50 arriving programs, program-aware keeps 20 Active + 30 Paused within budget; request-level would overrun it and thrash.

If your arriving program count is comfortably below fits, you do not have a thrashing problem yet and plain prefix caching suffices; the value of this recipe appears once concurrency crosses fits, which is exactly where a request scheduler starts paying the quadratic re-prefill cost the tech page's eviction model quantifies.

5. Wire the RL rollout path¶

For agentic RL, point your rollout engine's generation endpoint at the ThunderAgent port and set program_id per trajectory. The paper ships a SkyRL recipe that trains Qwen3-32B on R2E-Gym with SkyRL's fully asynchronous framework for a reported 3.01x wall-clock speedup, and a Search-R1 example on slime.

# Reference template: run ThunderAgent as the rollout serving layer, then launch
# the SkyRL/slime training loop pointed at :9000. See the repo's examples for the
# framework-specific launcher and config.
thunderagent --backend-type vllm --backends http://node1:8000 http://node2:8000 \
             --port 9000 --metrics
# then start the SkyRL rollout with its OpenAI base_url set to http://<host>:9000/v1

6. Watch the right metrics¶

Enable --metrics and track three things, not just throughput: the KV-cache hit rate (should stay high on deterministic-tool workloads; a lower rate with higher throughput is expected on stochastic ones), cross-node memory imbalance (the problem the global queue exists to bound; a persistent divergence means the queue is not rebalancing), and disk usage over time (should stay near-constant; linear growth means a client is not releasing completed programs).

Failure modes¶

Client never releases a program. Skipping POST /programs/release on completion leaves the sandbox and its disk pinned, re-creating the linear disk growth the manager exists to prevent. Make release part of the trajectory-teardown path, not a best-effort afterthought.
Sizing against peak context but running longer contexts. KV grows linearly with context length, so a rollout whose trajectories run longer than your AVG_CTX assumption holds fewer programs resident than planned; re-run the planner with the real p95 context length, not the mean.
One backend URL for a multi-node deployment. Passing a single --backends URL defeats the global waiting queue's cross-node balancing; list every replica so restores can route to the node with the most free capacity.
Version skew between ThunderAgent and the engine. ThunderAgent depends on the backend's memory accounting; a vLLM or SGLang upgrade can shift it. Pin both and re-validate the capacity numbers after any engine bump.
Reading a stochastic-workload dip as a regression. On workloads with unpredictable tool latency the scheduler trades hit rate for utilization and can sit below parity at low concurrency; judge it on throughput at your real concurrency, not on hit rate alone.

References¶

Kang, Li, Xu, Yang, Chen, Wang, Chen, Krishna, Xu, and Arora, "ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System" (arXiv 2602.13692): https://arxiv.org/abs/2602.13692
ThunderAgent repository (install, CLI flags, API, SkyRL/slime examples; MIT): https://github.com/ThunderAgent-org/ThunderAgent
vLLM documentation (backend engine, KV cache and prefix caching): https://docs.vllm.ai/
SGLang (alternative backend, RadixAttention prefix cache): https://github.com/sgl-project/sglang
SkyRL (asynchronous agentic RL framework): https://github.com/NovaSky-AI/SkyRL