Markdown

Program-aware agentic inference (ThunderAgent)¶

Scope: how to serve multi-turn agentic workflows, and run their RL rollouts, when the workload is a stream of interleaved LLM calls and tool calls rather than independent requests. A request-level serving stack (an LLM engine plus a separate tool orchestrator) schedules each call with no knowledge that a hundred of them belong to one workflow, so it thrashes the KV cache, imbalances memory across nodes, and leaks tool sandboxes. This page uses ThunderAgent (arXiv 2602.13692) as the worked example of the fix: abstract each workflow as an LLM Program and schedule its KV cache, cross-node placement, and tool environments as one unit. It is the serving-side companion to the agent harness (which owns the loop itself), to agentic and tool-use RL and async RL systems (which frame the rollout workload), and to prompt caching and KV cache management (the prefix-reuse mechanics it schedules over). The recipe for standing it up in front of vLLM is Cookbook: ThunderAgent agentic serving.

Numbers and quotes are from the ThunderAgent paper (arXiv 2602.13692v3, cs.OS) and its repository, read at research time; the ICML 2026 Spotlight acceptance is stated in the repository README, not on the arXiv PDF, so verify it before citing. The thunderagent CLI and API snippets are reference templates on the real interface (they need the installed package and a GPU backend); the Python scheduling model is executed and asserted (stdlib only).

flowchart TB
  subgraph CLIENTS["Agent clients (SWE-Agent, OpenHands, ToolOrchestra)"]
    A1["Workflow 1<br/>program_id=p1"]
    A2["Workflow N<br/>program_id=pN"]
  end
  A1 -->|"OpenAI-style call + program_id"| TA
  A2 -->|"OpenAI-style call + program_id"| TA
  subgraph TA["ThunderAgent middleware"]
    SCHED["Program-aware scheduler<br/>periodic thrashing monitor (every dt)<br/>Pause / Restore, shortest-context-first eviction<br/>global program-aware waiting queue"]
    RM["Tool resource manager<br/>hook-based GC on Terminated<br/>async environment preparation"]
  end
  SCHED -->|"route / pause / restore"| B1["vLLM / SGLang backend<br/>node 1 (KV cache)"]
  SCHED --> B2["vLLM / SGLang backend<br/>node M (KV cache)"]
  RM -->|"prepare / reclaim"| ENV["Tool environments<br/>Docker sandboxes, sockets (CPU cluster)"]
  B1 -.->|"reasoning tokens"| SCHED
  ENV -.->|"acting / tool results"| SCHED

What it is¶

An agentic program is one multi-turn workflow: an agent alternates reasoning turns (an LLM generates the next action) with acting turns (a tool runs and returns an observation), for as many rounds as the task needs. ThunderAgent's core move is to make that workflow a first-class scheduling unit instead of decomposing it into unrelated requests. The paper defines the unit formally as P = <ID, c, T, L, tau, s>: a global identifier, the context length c in tokens, the set of tool environments T the workflow needs, the backend placement L for cache affinity, the execution phase tau (Reasoning or Acting), and the scheduling status s (Active, Paused, or Terminated).¹ Because the unit persists across many model invocations and tool executions, the runtime can see the whole workflow: its growing KV cache, its live sandboxes, and whether it is currently on the GPU reasoning or off the GPU waiting on a tool.

The abstraction unifies two kinds of persistent state that request-level systems track separately. The KV cache of a program's context is the workflow's working memory, and keeping it resident across steps is what enables near-complete cache reuse from one turn to the next; the tool environment (a sandbox initialized on the first turn, holding open sockets and disk) must stay consistent and reachable for the workflow's entire life.¹ A stateless request scheduler sees neither: it treats turn 47's prompt as a fresh request and has no reason to keep turn 46's KV cache or the workflow's sandbox around.

ThunderAgent sits as middleware between agent clients and the infrastructure: it wraps existing LLM engines (vLLM, SGLang, TensorRT-LLM) through their OpenAI-style endpoints and a tool orchestrator, using the program_id on each call to attribute requests to workflows.² It is not a new inference engine; it is a program-aware scheduler and tool-resource manager layered over engines that already exist.

Why use it¶

The paper decomposes serving cost with a Belady-style space-time product, Cost_total ~ Cost_decode + Cost_prefill + Cost_recompute + Cost_unused + Cost_caching, where decode and prefill are useful work and the last three terms are pure overhead that a program-blind scheduler creates:³

KV cache thrashing (the recompute term). When the resident programs' contexts exceed a backend's capacity, a request scheduler evicts a workflow's KV cache, then re-prefills it from scratch on the workflow's next turn. The paper measures that re-prefill inflating average end-to-end latency by up to 7.14x versus a non-thrashing baseline.⁴
Cross-node memory imbalance (the unused term). Pinning every request from one workflow to the node it started on lets memory diverge across data-parallel replicas. In a 90-minute agentic RL rollout snapshot across two 8xH100 nodes, memory usage between the two nodes diverged by more than 20% for over 37 minutes and peaked at a 51% imbalance, so one node thrashed while the other sat underused.⁴
Tool lifecycle obliviousness (the caching term, plus disk). A scheduler that never learns a workflow ended never tears down its sandbox, so disk usage grows linearly with processed workflows until it exceeds capacity; and GPU memory sits idle holding a paused workflow's KV cache while its tool runs.⁴

Scheduling with the whole-workflow view attacks all three at once. The paper reports 1.48-3.58x serving throughput over vLLM and 1.17-3.31x over Continuum on an 8xH100 cluster at high concurrency, 1.79-3.92x RL rollout throughput over a vLLM plus SGLang Gateway baseline (OpenHands rollout rose from 69.1 to 270.8 steps per minute, mini-SWE-Agent from 375.4 to 671.8), and up to 4.2x disk-memory savings from the tool manager, which itself contributes about 10% of the latency win.⁷ Rollout is worth optimizing because, as the paper notes, it "accounts for over 70% of the total wall-clock time in reinforcement learning."⁸

When to use it (and when not)¶

Use program-aware serving when:

The workload is genuinely agentic: long multi-turn workflows whose per-turn context grows and whose KV cache should survive across turns, running at a concurrency high enough that resident contexts contend for backend capacity. This is where thrashing and imbalance actually bite.
You run RL rollouts for agentic tasks (SWE-Agent, OpenHands, tool-use agents), where rollout dominates wall-clock and each trajectory is a long-lived program with a sandbox. ThunderAgent has recipes for SkyRL and slime.¹¹
Tool environments are heavy (the paper's OpenHands sandboxes are over 10 GB each) or slow to initialize, so overlapping their setup with reasoning and reclaiming them promptly is a first-order cost.⁹

Do not reach for it when:

The workload is single-turn or stateless. With no cross-turn KV cache to preserve and no persistent sandbox, the LLM Program abstraction adds bookkeeping with nothing to schedule; a plain engine with prefix caching is simpler.
Concurrency is low enough that resident contexts never exceed backend capacity. Without contention there is no thrashing to prevent, and on the paper's own stochastic-tool-latency workload (ToolOrchestra/HLE), ThunderAgent's margin over the baselines shrinks to its smallest reported value (1.48x over vLLM); it still wins, but the win is narrower than on predictable-tool workloads.¹²
Tool-latency behavior is pathological in a way the runtime must predict. ThunderAgent deliberately avoids predicting tool durations (the paper argues tool latency is heavy-tailed with no stable central tendency, making prediction "intrinsically brittle") and instead reacts with dynamic cost modeling, but that means it holds some GPU memory idle during long tool calls rather than reclaiming it on a forecast.¹²

Architecture¶

The scheduler owns two primitive operations on a program: Restore admits a Paused program to a backend that has capacity, binding its KV cache and marking it Active; Pause unbinds an Active program, releasing its KV cache for preemption and marking it Paused.⁵ Everything else is policy over when to apply them.

A periodic thrashing monitor samples each backend's memory every dt and acts before an admitted workflow's context growth overruns capacity, in contrast to admission-only schedulers that check a workflow once on arrival and never again.⁵ When thrashing is imminent on a backend, the scheduler pauses programs to free dC = sum(c_p) - lambda_max * C_total tokens, choosing which to pause by shortest-context-first eviction: re-prefilling an evicted program costs quadratically in its context length (Lemma 4.1), so freeing a target amount by pausing many short workflows is far cheaper in future recompute than pausing one long one.⁵ A global program-aware waiting queue shared across all backend replicas replaces per-node pinning: because a paused program's KV cache is assumed evicted, its recompute cost is node-agnostic, so the scheduler can restore it onto whichever node has the most free capacity, bounding the unused term without sacrificing cache locality.⁵

The tool resource manager couples sandbox lifetime to the program's status through lifecycle hooks. When a program reaches Terminated, a hook triggers immediate teardown of its sandboxes, sockets, and compute slots, holding disk consumption near-constant instead of letting it grow with workflow count; and when a high-priority program approaches its restore threshold, the manager asynchronously prepares its execution environment before GPU memory is allocated, hiding tool-initialization latency behind reasoning for tool-heavy agents.⁶

How to use it¶

Integration is an OpenAI-compatible passthrough: run ThunderAgent in front of your existing engine, send requests to its port instead of the engine's, and tag each call with the workflow it belongs to. The paper states that only three changes are required.² Reference templates on the real interface (verify flags against the current repo):

# Reference template (repo README): serve a model, then front it with ThunderAgent.
uv pip install vllm --torch-backend=auto
vllm serve Qwen/Qwen3-32B --port 8000                      # the backend engine
thunderagent --backend-type vllm --backends http://localhost:8000 \
             --port 9000 --metrics --profile               # send requests to :9000

# Reference template (repo README): tag each call with its program_id so the
# scheduler can attribute the request to a workflow and reuse its KV cache.
extra_body = {"program_id": "unique_workflow_id"}
# extra_body["docker_ids"] = ["sandbox_a", "sandbox_b"]   # optional: tool assets
openai.client.chat.completions.create(
    model=model_name, messages=messages, extra_body=extra_body,
)
# On workflow completion, release so the hook-based GC reclaims the sandbox:
#   POST /programs/release  {"program_id": "unique_workflow_id"}

The program_id is the load-bearing field: it is what turns a stream of chat completions back into a schedulable program. The optional docker_ids list hands the tool manager the sandbox handles to reclaim on teardown.

How to develop with it¶

The scheduling logic worth internalizing before trusting a program-aware serving layer is the eviction policy, because it is what keeps the recompute term down, and it is small enough to validate directly. This executed model implements the thrashing check, the memory-to-free computation, and shortest-context-first eviction, then proves the property the paper's Lemma 4.1 rests on (many short evictions beat one long one under quadratic recompute cost), checks it against a brute-force optimum, and adversarially shows where the greedy heuristic is only near-optimal. Part B validates the pause and restore priority scores (the execution-phase indicator dominates the context term, so Acting programs are paused first and Reasoning programs restored first):

# thunder_sched.py -- validated: the two scheduling decisions ThunderAgent makes
# when a backend is about to thrash its KV cache. Pure stdlib.
#
# Part A: shortest-context-first eviction. Re-prefilling an evicted program's KV
# cache costs O(c^2) in its context length c (Lemma 4.1: the reprefill is a
# prefill over c tokens whose per-token attention is itself O(c)). To free a
# target amount of memory, evicting many SHORT programs is far cheaper than
# evicting one LONG one, because for a fixed freed amount DeltaC the cost of
# k equal programs is k*(DeltaC/k)^2 = DeltaC^2 / k, which shrinks as k grows.
#
# Part B: the Pause/Restore priority scores (Eq. 10, 11). The execution-phase
# indicator dominates the 1/c term, so ThunderAgent pauses Acting (off-GPU,
# tool-running) programs before Reasoning ones and restores Reasoning before
# Acting, breaking ties toward the shorter context.
from itertools import combinations


def recompute_cost(c: int) -> int:
    """Lemma 4.1: reprefilling a c-token context costs O(c^2)."""
    return c * c


# --- Part A1: the convexity core -- freeing a fixed amount with more, shorter
# programs is strictly cheaper (DeltaC^2 / k, monotically decreasing in k) -------
DELTA_C = 1200
costs_by_count = [k * recompute_cost(DELTA_C // k) for k in (1, 2, 4, 8) if DELTA_C % k == 0]
assert costs_by_count == sorted(costs_by_count, reverse=True), costs_by_count
# one 1200-token program costs 1_440_000; eight 150-token programs cost 180_000: 8x cheaper.
assert recompute_cost(1200) == 1_440_000
assert 8 * recompute_cost(150) == 180_000
assert 8 * recompute_cost(150) * 8 == recompute_cost(1200)   # exactly DeltaC^2/k scaling


def thrash_free_target(capacity: int, contexts: list[int], lam_max: float = 1.0) -> float:
    """Eq. 6 thrashing check + the memory to free. A backend thrashes when its
    capacity is below the resident context sum; free down to lam_max*capacity."""
    resident = sum(contexts)
    if resident <= lam_max * capacity:
        return 0.0                      # not thrashing, nothing to evict
    return resident - lam_max * capacity


def evict_greedy(contexts: list[int], to_free: float, longest_first: bool = False) -> list[int]:
    """Greedily pause/evict programs until >= to_free tokens are released.
    Shortest-context-first is ThunderAgent's stated strategy; longest_first is
    the natural alternative it is compared against."""
    freed, chosen = 0, []
    for c in sorted(contexts, reverse=longest_first):
        if freed >= to_free:
            break
        chosen.append(c)
        freed += c
    return chosen


def brute_force_min_cost(contexts: list[int], to_free: float) -> int:
    """Smallest achievable total recompute cost over every subset that frees
    >= to_free tokens (reference optimum, exponential; small inputs only)."""
    best = None
    for r in range(1, len(contexts) + 1):
        for combo in combinations(contexts, r):
            if sum(combo) >= to_free:
                cost = sum(recompute_cost(c) for c in combo)
                best = cost if best is None else min(best, cost)
    return best


# --- Part A2: realistic thrashing -- many short agentic programs plus a few long
# ones. Shortest-first frees the memory at a fraction of longest-first's cost. ---
CAP = 4000
CTX = [120, 150, 180, 200, 220, 900, 1100]      # 5 short agents, 2 long ones
assert sum(CTX) == 2870 <= CAP                    # no thrash yet
CTX_THRASH = CTX + [1400]                          # an 8th program tips it over
need = thrash_free_target(CAP, CTX_THRASH)
assert need == sum(CTX_THRASH) - CAP == 270        # free 270 tokens

short_pick = evict_greedy(CTX_THRASH, need)
long_pick = evict_greedy(CTX_THRASH, need, longest_first=True)
short_cost = sum(recompute_cost(c) for c in short_pick)
long_cost = sum(recompute_cost(c) for c in long_pick)
# shortest-first evicts {120,150} = 270 tokens, cost 36_900; longest-first evicts
# {1400} = 1400 tokens, cost 1_960_000: shortest-first is ~53x cheaper here.
assert short_pick == [120, 150] and short_cost == 36_900
assert long_cost == 1_960_000
assert short_cost < long_cost
# and shortest-first is the true optimum on this instance (short programs tile DeltaC).
assert short_cost == brute_force_min_cost(CTX_THRASH, need)

# --- Part A3: adversarial honesty -- greedy shortest-first is a heuristic, not a
# global optimum. When one program exactly matches DeltaC, greedy overshoots and
# is beaten by the brute-force pick, so the strategy is near-optimal, not optimal.
PATHO = [1, 1, 10]
greedy = evict_greedy(PATHO, to_free=10)                  # {1,1,10} overshoots
greedy_cost = sum(recompute_cost(c) for c in greedy)
opt = brute_force_min_cost(PATHO, to_free=10)             # {10} alone
assert greedy == [1, 1, 10] and greedy_cost == 102
assert opt == 100 and greedy_cost > opt                   # greedy is worse here


# --- Part B: Pause/Restore scores. Phase indicator dominates the 1/c term -------
def s_pause(c: int, phase: str) -> float:      # Eq. 11: pause Acting first
    return 1.0 / c + (1.0 if phase == "A" else 0.0)


def s_restore(c: int, phase: str) -> float:    # Eq. 10: restore Reasoning first
    return 1.0 / c + (1.0 if phase == "R" else 0.0)


# Programs mixing phases and context lengths (contexts >= 2 so 1/c <= 0.5 and the
# indicator strictly separates phases).
progs = [("p_act_long", 1000, "A"), ("p_act_short", 40, "A"),
         ("p_reason_long", 1200, "R"), ("p_reason_short", 30, "R")]

pause_order = [name for name, c, ph in sorted(progs, key=lambda x: s_pause(x[1], x[2]), reverse=True)]
restore_order = [name for name, c, ph in sorted(progs, key=lambda x: s_restore(x[1], x[2]), reverse=True)]

# Pause: both Acting programs come before both Reasoning ones; shorter first within phase.
assert pause_order == ["p_act_short", "p_act_long", "p_reason_short", "p_reason_long"]
# Restore: both Reasoning programs first; shorter first within phase.
assert restore_order == ["p_reason_short", "p_reason_long", "p_act_short", "p_act_long"]
# The indicator (1.0) always outweighs the context term (1/c <= 0.5), so phase wins.
assert s_pause(1000, "A") > s_pause(2, "R")

print(f"OK Part A: fixed-DeltaC cost {costs_by_count} shrinks with more/shorter evictions; "
      f"thrash frees {need:.0f} tok, shortest-first cost {short_cost} vs longest-first "
      f"{long_cost} (={long_cost // short_cost}x), == brute-force optimum {brute_force_min_cost(CTX_THRASH, need)}. "
      f"Honesty: on {PATHO} greedy={greedy_cost} > optimum={opt}. "
      f"OK Part B: pause {pause_order[:2]} (Acting) first, restore {restore_order[:2]} (Reasoning) first.")

Run output:

OK Part A: fixed-DeltaC cost [1440000, 720000, 360000, 180000] shrinks with more/shorter evictions; thrash frees 270 tok, shortest-first cost 36900 vs longest-first 1960000 (=53x), == brute-force optimum 36900. Honesty: on [1, 1, 10] greedy=102 > optimum=100. OK Part B: pause ['p_act_short', 'p_act_long'] (Acting) first, restore ['p_reason_short', 'p_reason_long'] (Reasoning) first.

The honesty caveat matters operationally: greedy shortest-first is the paper's stated strategy and it is optimal when many short programs tile the target, which is the realistic high-concurrency thrashing regime, but it can overshoot when one program's context happens to match the free target exactly. Treat it as a fast near-optimal heuristic, not a knapsack solver.

How to run it in production¶

Tune dt and the priority decay to your workload. The paper runs the monitor interval dt = 5 and an acting-token priority decay f(t) = 2^-t, and sets both watermarks lambda_max and lambda_min to 1 because the shared prompt across programs already reserves a memory buffer.⁹ The decay is not a free parameter: the paper proves that if tool-execution latency is memoryless, exponential decay is the only admissible form, so keep it exponential and tune the base.⁵
Split the GPU and tool tiers. The evaluation runs LLM engines on the GPU cluster and offloads agent Docker environments to a dedicated CPU cluster.⁹ Co-locating heavy sandboxes on GPU nodes wastes the memory the scheduler is trying to balance.
Watch the KV-cache hit rate, not just throughput. ThunderAgent holds a near-100% hit rate on deterministic-tool workloads where a request scheduler's collapses from over 90% to about 60% under concurrency, but on stochastic workloads it deliberately accepts a lower hit rate for higher utilization, so hit rate alone can look worse while throughput is better.⁷¹²
Verify it scales the way you deploy. The reported speedup over an SGLang cache-aware baseline widens from 1.79x at two nodes to 2.39x at eight (64xH100), and the global scheduler's per-tick communication delay grows sub-linearly (13.5 ms at two nodes to 22.7 ms at eight), so the central scheduler is not the bottleneck at that scale, but re-measure at yours.¹⁰
Pin versions across the stack. ThunderAgent wraps a specific engine (vLLM or SGLang) through its OpenAI endpoint; an engine upgrade can change memory accounting the scheduler depends on. It is also being standardized into NVIDIA Dynamo 2.0 as the nvext.agent_context serving protocol, so the integration surface is still moving.¹¹

Failure modes¶

Thrashing avoidance idling GPU memory. Guaranteeing no thrashing means a paused acting program's freed memory can sit unused while its long tool call runs; the exponential time-decay f(t) mitigates this but does not eliminate it, so a workload of very long tool calls can leave the GPU under-utilized.¹²
Assuming the speedup is uniform across workloads. On the paper's predictable-tool workloads ThunderAgent's margin over vLLM and Continuum reaches 2.43-3.58x; on its stochastic-tool-latency workload the margin narrows to 1.48x. It still wins in every reported configuration, but do not assume the largest reported multiplier applies to your own workload's tool-latency profile.¹²
Program leaks if release is skipped. The hook-based GC reclaims a sandbox on Terminated status; a client that never marks a workflow complete (crash, forgotten POST /programs/release) leaves the sandbox and its disk pinned, re-creating the linear disk growth the manager exists to prevent. Treat release as mandatory, not best-effort.
Reported figures with internal inconsistencies. The paper's own Figure 4 caption says up to 2.43-3.56x while the body says up to 3.58x, and the "40 H100" figure for the SkyRL 3x rollout speedup appears only in the README, not the paper body (which reports 3.01x without the GPU count). Cite the body ranges (1.48-3.58x serving, 1.79-3.92x rollout) and treat figure-embedded per-panel numbers as approximate.⁷¹¹
Assuming the abstraction helps a stateless workload. Tagging single-turn, sandbox-free requests with program_id adds scheduling state with nothing to preserve across turns; the win requires long-lived programs with resident KV cache and persistent tools.

References¶

Kang, Li, Xu, Yang, Chen, Wang, Chen, Krishna, Xu, and Arora, "ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System" (arXiv 2602.13692, cs.OS): https://arxiv.org/abs/2602.13692
ThunderAgent repository (source, install, API, integrations; MIT): https://github.com/ThunderAgent-org/ThunderAgent
ThunderAgent blog: https://thunderagent.ai
vLLM (backend inference engine): https://github.com/vllm-project/vllm
SGLang (backend inference engine, RadixAttention prefix cache): https://github.com/sgl-project/sglang
SkyRL (asynchronous agentic RL framework with a ThunderAgent recipe): https://github.com/NovaSky-AI/SkyRL
NVIDIA Dynamo (serving framework integrating the program abstraction as nvext.agent_context): https://github.com/ai-dynamo/dynamo

ThunderAgent (arXiv 2602.13692). Section 4.1 (Eq. 1) formalizes an agentic program as the tuple P = <ID, c, T, L, tau, s> (identifier, context tokens, tool environments, backend placement, execution phase Reasoning/Acting, status Active/Paused/Terminated); Section 1 describes the same abstraction as "a first-class scheduling unit that persists across multiple model invocations and tool executions, exposing semantic state to the runtime," decoupling scheduling from the execution backend (vLLM/SGLang/TensorRT-LLM). Two persistent states (Section 2.1): GPU memory (the KV cache is the workflow's memory, enabling near-complete reuse across steps) and the tool environment (sandboxes initialized at the first turn must stay consistent and reachable throughout). ↩↩
ThunderAgent Section 4.1 and repository README: "ThunderAgent directly wraps existing LLM engines and tool orchestrators by interfacing with OpenAI-style endpoints. Program IDs allow the system to distinguish requests from different agentic workflows." The README describes it as sitting "between agent clients and the infrastructure layer as an agentic workflow scheduler." Integration is three changes (Appendix B): add program_id (and optional docker_ids) to each OpenAI call via extra_body, pass program_id into tool execution, and POST /programs/release on completion. ↩↩
ThunderAgent Section 4.2: cost is a Belady space-time product Cost_x = integral of M_x(t) dt; total decomposes (Eq. 3) as Cost_total ~ Cost_decode + Cost_prefill + Cost_recompute + Cost_unused + Cost_caching, where decode and prefill are effective work and recompute (KV thrashing), unused (memory imbalance), and caching (idle GPU memory during tool execution) are wasted system overhead. ↩
ThunderAgent Sections 1 and 3: (1) KV cache thrashing re-prefill "increases the average end-to-end latency of agent workflows by up to 7.14x" versus a non-thrashing setting; (2) profiling agentic RL rollout on GLM-4.6 across two 8xH100 data-parallel nodes, "the memory usage between two DP nodes diverges by more than 20% for over 37 minutes, reaching a peak imbalance of 51%" during a 90-minute snapshot; (3) tool-lifecycle obliviousness grows disk usage linearly with processed workflows until it exceeds capacity. ↩↩↩
ThunderAgent Section 4.3: Restore (Eq. 4) admits a Paused program to a backend with capacity; Pause (Eq. 5) unbinds an Active program and releases its KV cache. A periodic monitor checks memory every dt (versus admission-only checks in Continuum); on imminent thrashing (Eq. 6, C_total < sum(c_p)) it frees dC = sum(c_p) - lambda_max * C_total. Lemma 4.1: Cost_recompute scales quadratically with context length c_i, so eviction minimizes sum(c_i^2) subject to sum(c_i) >= dC (Def. 4.1), giving greedy shortest-context-first eviction. Restore/pause scores S_restore = 1/c + I(tau=R) (Eq. 10) and S_pause = 1/c + I(tau=A) (Eq. 11). A global program-aware waiting queue makes recompute cost node-agnostic for paused programs, bounding the unused cost. The time-decay of acting-token weight (Eq. 7) must be exponential: the paper proves exponential decay is the only admissible f(t) when tool latencies are memoryless (Appendix F.1). ↩↩↩↩↩
ThunderAgent Section 4.4: hook-based garbage collection couples resource persistence to program status; on Terminated "the collector triggers an immediate teardown sequence, systematically reclaiming sandboxes, network sockets, and compute slots," holding disk consumption near-constant. Asynchronous environment preparation restores a high-priority program's execution environment before GPU memory is allocated, "effectively hides the initialization overhead," overlapping I/O-intensive setup with LLM reasoning. ↩
ThunderAgent Section 5 and abstract: serving throughput 1.48-3.58x over vLLM and 1.17-3.31x over Continuum on 8xH100 at high concurrency (abstract rounds to 1.5-3.6x); RL rollout 1.79-3.92x over vLLM+Gateway (Table 2, GLM-4.6, N=144, 2xH100: mini-SWEAgent 375.4 to 671.8 steps/min, OpenHands 69.1 to 270.8); up to 4.2x disk-memory savings with the tool policy contributing about 10% of the latency improvement (Section 5.4). KV-cache hit rate stays near 100% on deterministic-tool workloads (Mini-SWE-Bench, OpenHands) where Continuum's drops from over 90% to about 60% under concurrency. Internal inconsistency: Figure 4 caption says up to 2.43-3.56x, the body says up to 3.58x. ↩↩↩
ThunderAgent Section 1: "rollout accounts for over 70% of the total wall-clock time in reinforcement learning (RL)," motivating the rollout-throughput target. ↩
ThunderAgent Section 5.1: models GLM-4.6 (355B) and Qwen3-235B quantized to FP8 with tensor parallelism TP8 on 8xH100 nodes; ToolOrchestra runs Qwen3-8B FP16 on one RTX 5090; hyperparameters dt = 5 and priority decay f(t) = 2^-t; watermarks lambda_max = lambda_min = 1; LLM engines on GPU clusters and agent Docker environments offloaded to a dedicated CPU cluster; OpenHands sandboxes exceed 10 GB each, mini-SWEAgent about 2 GB. Baselines: vLLM (request-aware, stateless), Continuum (SOTA multi-turn, predicts tool durations and pins KV cache to HBM), and vLLM plus SGLang Gateway (distributed RL rollout). ↩↩↩
ThunderAgent Appendix A.3: on GLM-4.6 with mini-SWEAgent at 72 programs per node, speedup over the SGLang Model Gateway cache-aware routing baseline widens from 1.79x at 2 nodes to 2.39x at 8 nodes (64xH100); the global scheduler's per-tick communication delay grows sub-linearly from 13.5 ms at 2 nodes to 22.7 ms at 8 nodes (1.68x for a 4x node increase). ↩
ThunderAgent Appendix A.6 and README: a SkyRL recipe "trains Qwen3-32B on R2EGym with SkyRL's fully asynchronous framework and achieves a 3.01x wall-clock speedup over the baseline" (the README states 3x on 40 H100 GPUs, a figure not printed in the paper body); a Search-R1 plus slime RL example; and integration into NVIDIA Dynamo 2.0, where the program abstraction "is standardized as part of the nvext.agent_context serving protocol." The ICML 2026 Spotlight (top 2.2%) acceptance is stated in the README News, not on the arXiv PDF. ↩↩↩
ThunderAgent Appendix C and Section 5.2: tool latency "lacks a stable central tendency; instead, heavy-tailed behavior dominates, making tool latency prediction intrinsically brittle in practice," so ThunderAgent uses reactive dynamic cost modeling rather than duration prediction; guaranteeing no thrashing means acting programs' GPU memory can idle during long tool calls (mitigated by f(t)); and on stochastic workloads it "exhibits a lower KV cache hit rate than Continuum" while achieving higher throughput, with some low-concurrency points below parity (about 0.65x on ToolOrchestra/HLE). There is no dedicated Limitations section in the paper; these are the nearest explicit constraint statements. ↩↩↩↩↩