Markdown

Cookbook: ThunderAgent agentic serving in front of vLLM¶

Scope: the concrete deployment recipe for program-aware agentic inference, from a running vLLM backend to an agent client whose multi-turn calls the scheduler can keep resident. This page is the operational how-to: the commands to launch the middleware, the three client-side changes, an executed KV-cache capacity planner that tells you how many concurrent programs a backend holds before it thrashes, and the wiring for an RL rollout. The concepts, the pause/restore scheduler, and the eviction proof live on the tech page; the rollout-workload framing is agentic RL and async RL systems.

The vllm, thunderagent, and client snippets are reference templates on the real interfaces (they need the installed packages and a GPU backend, and flags move between versions, so verify against the current repo). The Python capacity planner is executed and asserted (stdlib only); its per-model layer and head counts are illustrative placeholders you must replace with your model's real config.

flowchart LR
  subgraph SETUP["Deploy once"]
    S1["1. vLLM serves the model<br/>:8000"] --> S2["2. thunderagent fronts it<br/>:9000"]
  end
  S2 --> S3["3. Client sends to :9000<br/>with program_id"]
  S3 --> S4["4. Size concurrency<br/>KV capacity planner"]
  S4 --> S5["5. Run rollout<br/>(SkyRL / slime)"]
  S5 --> S6["6. Watch metrics<br/>hit rate, imbalance, disk"]
  S6 -.->|"tune dt, decay,<br/>node count"| S2

What it is¶

A recipe for inserting ThunderAgent as a scheduling and tool-management layer between your agent clients and one or more inference backends, without replacing the backend. You keep vLLM (or SGLang) exactly as you run it today; ThunderAgent listens on a new port, forwards requests to the backend, and uses a program_id on each call to schedule whole workflows instead of isolated requests. The payoff, the mechanics, and the reported speedups are on the tech page; this page is the sequence of steps and the sizing math.

Why use it¶

You already have a vLLM or SGLang deployment and want the program-aware wins (fewer KV re-prefills, balanced memory across nodes, reclaimed tool sandboxes) without rewriting the serving stack. The integration is three client-side changes and one extra process.
Your rollouts thrash at high concurrency. When many long agentic trajectories run at once, their combined KV cache exceeds a backend's capacity and a request scheduler evicts and re-prefills; sizing concurrency against the planner below tells you when that happens and how pausing avoids it.
Your tool sandboxes are heavy or leaky. The tool manager reclaims sandboxes on workflow completion and prepares them asynchronously, which matters most when a sandbox is gigabytes and slow to start.

When to use it (and when not)¶

Use this recipe for multi-turn agentic serving or agentic RL rollout at concurrency high enough that resident contexts contend for backend memory. Skip it for single-turn or stateless serving (nothing persists across turns to schedule), and re-measure before trusting a speedup on stochastic-tool workloads, where the tech page documents sub-parity points. If you are not yet contending for KV capacity, the capacity planner below will tell you: if your arriving program count is well under fits, plain vLLM with prefix caching is enough.

Architecture¶

Three planes stay separate. The backend plane is the unchanged vLLM or SGLang engine holding model weights and the KV cache. The scheduling plane is ThunderAgent: it terminates the OpenAI-style client connection, attributes each call to a program, and issues pause/restore against the backend. The tool plane is the sandbox fleet (Docker containers on a CPU cluster in the paper's deployment) that the resource manager prepares and reclaims. The client talks only to the scheduling plane's port.

How to use it¶

1. Serve the model with a backend engine¶

Bring up vLLM as you normally would; ThunderAgent does not change how the model is loaded or quantized.

# Reference template. Serve the policy model on the backend port.
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen3-32B --port 8000 \
  --tensor-parallel-size 8            # match your GPU count / model size

2. Front it with ThunderAgent¶

Install ThunderAgent and point it at the backend. Clients will send to port 9000 from here on.

# Reference template (repo README).
git clone https://github.com/ThunderAgent-org/ThunderAgent.git
cd ThunderAgent && pip install -e .
thunderagent --backend-type vllm --backends http://localhost:8000 \
             --port 9000 --metrics --profile
# --metrics/--profile expose the per-program token, tool-time, and hit-rate views.

For a multi-node backend, pass several --backends URLs so the global program-aware waiting queue can balance restores across replicas.

3. Tag client calls with program_id (the three changes)¶

The only client edits: set base_url to the ThunderAgent port, add program_id (and optional docker_ids) to each call, and release the program when the workflow ends so the GC reclaims its sandbox.

# Reference template (repo README + paper Appendix B). Three changes total. Requires the
# `openai` package and a running ThunderAgent, so this block is not executed here. Note the
# README writes the call as `openai.client.chat.completions.create(...)`; the SDK exposes no
# module-level `client`, so construct it explicitly as below.
import openai
client = openai.OpenAI(base_url="http://localhost:9000/v1", api_key="x")  # (1) port

def agent_turn(messages, program_id, sandbox_ids):
    extra_body = {"program_id": program_id}          # (2) attribute to a workflow
    if sandbox_ids:
        extra_body["docker_ids"] = sandbox_ids        #     hand over tool assets
    return client.chat.completions.create(
        model="Qwen/Qwen3-32B", messages=messages, extra_body=extra_body,
    )

# (3) On completion: POST /programs/release {"program_id": program_id}
#     so the hook-based GC tears down the sandbox and frees disk.

4. Size concurrency with the KV-cache capacity planner¶

Before you launch a large rollout, work out how many programs a backend can hold resident before it thrashes. That number, fits, is the concurrency at which a request-level scheduler starts evicting and re-prefilling; ThunderAgent instead pauses the excess. This planner is executed and self-checking; replace the illustrative model config with your model's real layer and head counts:

# thunder_capacity.py -- validated: size how many concurrent agentic programs a
# backend can hold before its KV cache thrashes, and show what program-aware
# pausing buys over a request-level scheduler that just OOMs. Pure stdlib.
#
# The KV-cache formula is standard and verifiable; the specific layer/head numbers
# below are ILLUSTRATIVE placeholders -- plug your own model's config.

GiB = 1024 ** 3


def kv_bytes_per_token(n_layers: int, n_kv_heads: int, head_dim: int, dtype_bytes: int) -> int:
    """Resident KV cache per context token: K and V (hence 2x), for every layer,
    across the key/value attention heads. This is the standard accounting an
    inference engine uses to budget its cache."""
    return 2 * n_layers * n_kv_heads * head_dim * dtype_bytes


def kv_token_budget(hbm_bytes: int, weight_bytes: int, reserve_frac: float = 0.10) -> float:
    """Tokens of KV cache a backend can hold: HBM minus weights minus a reserve
    for activations/fragmentation, divided by bytes-per-token happens in caller."""
    free = hbm_bytes * (1.0 - reserve_frac) - weight_bytes
    return max(0.0, free)


def max_resident_programs(free_kv_bytes: float, bytes_per_tok: int, avg_context: int) -> int:
    """How many programs of avg_context tokens fit resident at once."""
    if avg_context <= 0:
        return 0
    return int(free_kv_bytes // (bytes_per_tok * avg_context))


# --- Illustrative model + hardware (replace with your model's real config) ------
LAYERS, KV_HEADS, HEAD_DIM = 64, 8, 128     # a GQA-style ~30B-class placeholder
BF16 = 2
bpt = kv_bytes_per_token(LAYERS, KV_HEADS, HEAD_DIM, BF16)
assert bpt == 2 * 64 * 8 * 128 * 2 == 262_144   # 256 KiB / token

HBM = 80 * GiB          # one H100-80GB
WEIGHTS = 32 * GiB      # ~32 GiB of FP8/BF16 weights on this device (illustrative)
free = kv_token_budget(HBM, WEIGHTS)
resident_tokens = free / bpt
AVG_CTX = 8000          # a mid-rollout agentic context

fits = max_resident_programs(free, bpt, AVG_CTX)
# ~40 GiB of KV budget / 256 KiB per token ~ 163k resident tokens ~ 20 programs.
assert 18 <= fits <= 22, fits

# --- Sanity cross-checks on the formula -----------------------------------------
# bf16 KV is exactly half of fp32 KV (dtype linearity).
assert kv_bytes_per_token(LAYERS, KV_HEADS, HEAD_DIM, 4) == 2 * bpt
# Monotonic: more HBM never reduces the resident program count.
assert max_resident_programs(kv_token_budget(96 * GiB, WEIGHTS), bpt, AVG_CTX) >= fits
# Boundary: weights that fill HBM leave no KV budget and zero programs, no crash.
assert max_resident_programs(kv_token_budget(HBM, HBM), bpt, AVG_CTX) == 0
# Longer contexts hold fewer programs (KV grows linearly with context length).
assert max_resident_programs(free, bpt, 2 * AVG_CTX) <= fits // 2 + 1

# --- Program-aware pausing vs request-level OOM ---------------------------------
# A request-level scheduler admits all N arriving programs; once resident tokens
# exceed the budget it thrashes (evict + reprefill) or OOMs. ThunderAgent instead
# keeps at most `fits` programs Active and Pauses the rest, so N programs make
# progress on a backend that can only hold `fits` of them resident at once.
def active_and_paused(arriving: int, capacity_programs: int) -> tuple[int, int]:
    active = min(arriving, capacity_programs)
    return active, max(0, arriving - active)

N = 50
active, paused = active_and_paused(N, fits)
assert active == fits and paused == N - fits          # e.g. 20 active, 30 paused
# The paused programs' KV is evicted, so the resident token count stays within
# budget instead of overflowing it -- this is what avoids the thrash.
assert active * AVG_CTX <= resident_tokens

print(f"OK: {bpt // 1024} KiB/token; {free / GiB:.1f} GiB KV budget holds "
      f"~{resident_tokens / 1000:.0f}k tokens = {fits} programs of {AVG_CTX} tok resident. "
      f"For {N} arriving programs, program-aware keeps {active} Active + {paused} Paused "
      f"within budget; request-level would overrun it and thrash.")

Run output:

OK: 256 KiB/token; 40.0 GiB KV budget holds ~164k tokens = 20 programs of 8000 tok resident. For 50 arriving programs, program-aware keeps 20 Active + 30 Paused within budget; request-level would overrun it and thrash.

If your arriving program count is comfortably below fits, you do not have a thrashing problem yet and plain prefix caching suffices; the value of this recipe appears once concurrency crosses fits, which is exactly where a request scheduler starts paying the quadratic re-prefill cost the tech page's eviction model quantifies.

5. Wire the RL rollout path¶

For agentic RL, point your rollout engine's generation endpoint at the ThunderAgent port and set program_id per trajectory. The paper ships a SkyRL recipe that trains Qwen3-32B on R2E-Gym with SkyRL's fully asynchronous framework for a reported 3.01x wall-clock speedup, and a Search-R1 example on slime.

# Reference template: run ThunderAgent as the rollout serving layer, then launch
# the SkyRL/slime training loop pointed at :9000. See the repo's examples for the
# framework-specific launcher and config.
thunderagent --backend-type vllm --backends http://node1:8000 http://node2:8000 \
             --port 9000 --metrics
# then start the SkyRL rollout with its OpenAI base_url set to http://<host>:9000/v1

6. Watch the right metrics¶

Enable --metrics and track three things, not just throughput: the KV-cache hit rate (should stay high on deterministic-tool workloads; a lower rate with higher throughput is expected on stochastic ones), cross-node memory imbalance (the problem the global queue exists to bound; a persistent divergence means the queue is not rebalancing), and disk usage over time (should stay near-constant; linear growth means a client is not releasing completed programs).

How to develop with it¶

Local iteration differs from the deployment above in one way that matters: a single-node backend with a small model hides every bug the global waiting queue exists to catch, so develop against the smallest configuration that still exercises the scheduler.

Reproduce thrashing deliberately before you trust the scheduler. Set the backend's KV budget deliberately low (a small --gpu-memory-utilization on vLLM) and drive more concurrent programs at it than the planner in step 4 says fit. A correct setup shows programs entering Paused and later Restored while resident tokens stay inside the budget; a setup that instead OOMs or stalls means requests are reaching the engine without a program_id and are being scheduled request-at-a-time.
Assert the tag survives the client path. The commonest integration bug is an extra_body that a wrapper library drops silently, which fails open: every request becomes its own program, the hit rate collapses, and nothing errors. Log program_id at the gateway and assert it is non-null on a sample of requests before benchmarking anything.
Keep two backends in the dev loop. A single --backends URL cannot exercise cross-node restore routing, so a balancing bug only appears in production. Two small replicas on one host are enough to catch it.
Validate the scheduling logic in isolation. The eviction and pause/restore rules are small enough to model directly; the executed block on program-aware agentic inference implements them with assertions, including the case where the greedy heuristic is beaten by a brute-force optimum.

How to run it in production¶

Split the GPU and tool tiers. Run the engines on GPU nodes and the agent sandboxes on a separate CPU pool. Co-locating heavy Docker sandboxes on GPU nodes consumes the very memory the scheduler is balancing.
List every backend replica. The global waiting queue balances restores across the replicas it knows about; a single --backends URL silently disables that.
Size admission from the planner, not from throughput. Step 4's resident-program count is the admission ceiling. Admitting past it does not fail loudly, it thrashes: the engine evicts and re-prefills, and the recompute cost is quadratic in context length.
Alert on disk growth and on orphaned programs. Near-constant disk is the healthy signal. Linear growth means clients are not releasing, and the sandbox GC never sees a Terminated status to act on.

How to maintain it¶

Re-run the capacity planner after any model, dtype, or engine change. Every input to the KV-bytes-per-token formula (layers, KV heads, head dimension, cache dtype) is a version-coupled fact. An FP8-KV rollout or a model swap changes the resident-program count, and the old number will look fine right up to the point it thrashes.
Pin ThunderAgent and the backend engine together. ThunderAgent depends on the backend's memory accounting; a vLLM or SGLang bump can shift it. Re-validate the capacity numbers after either moves, not just after a ThunderAgent upgrade.
Track the upstream protocol. ThunderAgent is being standardized into NVIDIA Dynamo 2.0 as the nvext.agent_context serving protocol, so program_id and docker_ids are a moving integration surface. A renamed field fails open rather than erroring, which is exactly the failure that shows up as a quiet hit-rate regression weeks later.
Re-check the p95 context length on a schedule. Trajectories lengthen as agents gain tools. The planner sized against the context length you had, and KV grows linearly in it, so a drifting p95 quietly erodes the resident-program count you provisioned for.

Failure modes¶

Client never releases a program. Skipping POST /programs/release on completion leaves the sandbox and its disk pinned, re-creating the linear disk growth the manager exists to prevent. Make release part of the trajectory-teardown path, not a best-effort afterthought.
Sizing against peak context but running longer contexts. KV grows linearly with context length, so a rollout whose trajectories run longer than your AVG_CTX assumption holds fewer programs resident than planned; re-run the planner with the real p95 context length, not the mean.
One backend URL for a multi-node deployment. Passing a single --backends URL defeats the global waiting queue's cross-node balancing; list every replica so restores can route to the node with the most free capacity.
Version skew between ThunderAgent and the engine. ThunderAgent depends on the backend's memory accounting; a vLLM or SGLang upgrade can shift it. Pin both and re-validate the capacity numbers after any engine bump.
Reading a stochastic-workload dip as a regression. On workloads with unpredictable tool latency the scheduler trades hit rate for utilization and can sit below parity at low concurrency; judge it on throughput at your real concurrency, not on hit rate alone.

References¶

Kang, Li, Xu, Yang, Chen, Wang, Chen, Krishna, Xu, and Arora, "ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System" (arXiv 2602.13692): https://arxiv.org/abs/2602.13692
ThunderAgent repository (install, CLI flags, API, SkyRL/slime examples; MIT): https://github.com/ThunderAgent-org/ThunderAgent
vLLM documentation (backend engine, KV cache and prefix caching): https://docs.vllm.ai/
SGLang (alternative backend, RadixAttention prefix cache): https://github.com/sgl-project/sglang
SkyRL (asynchronous agentic RL framework): https://github.com/NovaSky-AI/SkyRL