Markdown

Sizing the agentic rollout sandbox fleet¶

Scope: the container fleet that agentic RL rollouts run inside, and how many machines it takes. When an RL task is a terminal workspace rather than a prompt, every rollout allocates a sandbox, builds an image, holds it for the length of a multi-turn episode, and runs a verifier at the end. This page derives the concurrency, node count, storage, token demand, and failure budget for that fleet from 327,189 measured agent trajectories, and shows why the GPU is nearly idle while the fleet is full. The training loop that consumes these rollouts is agentic and tool-use RL, the GPU-side pool ratio is rollout fleet sizing, and the tasks themselves come from verified task synthesis.

Evidence status, verified 2026-08-07. Every measurement below was computed here from two public CC BY 4.0 metadata parquets: Recursive-Task-Synthesis-Trajectories/metadata/trajectories.parquet (327,189 rows, 19.9 MB) and Recursive-Task-Synthesis/metadata/tasks.parquet (37,484 rows, 124 MB), released alongside arXiv 2608.05466. Concurrency is a sweep line over the started_at and finished_at fields; token and turn statistics come from input_tokens, output_tokens, and episode_count. The NumPy block was executed and its assertions pass. This is one campaign on one provider with one harness (Terminus-2 on Harbor, Daytona sandboxes), so the shape of the result should transfer and the constants should be re-measured on your own fleet. No cluster was provisioned here and no throughput was reproduced.

flowchart TB
  TRAINER["Trainer pool<br/>GPU, waits on batches"] --> ORCH["RL orchestrator<br/>hands out task ids"]
  ORCH --> Q["Sandbox admission queue"]
  Q --> POOL["Sandbox fleet<br/>1 vCPU / 2 GB / 6 GB per slot<br/>CPU-bound: ~1 slot per core"]
  POOL --> BUILD["Build image<br/>median 600 s budget"]
  BUILD --> EP["Agent episode<br/>median 11 turns, 22 min hold"]
  EP -->|every turn| GEN["Inference pool<br/>prefill 48 tok/s per sandbox<br/>decode 1.7 tok/s per sandbox"]
  GEN --> EP
  EP --> VER["Verifier run<br/>binary reward"]
  VER -->|reward| ORCH
  VER -->|29.4% exception| DROP["Dropped, ~1% ever pass"]

What it is¶

A rollout sandbox fleet is the pool of isolated containers an agentic RL loop needs, one per in-flight trajectory. It is a distinct resource from the trainer GPUs and from the inference GPUs, it is usually CPU and disk rather than accelerator, and it is the resource that quietly sets the ceiling on rollout throughput once tasks stop being prompts and start being workspaces.

The unit is a task bundle instantiated in a fresh container: build the image, drop the agent into the workspace with the public instruction, let it run commands for some number of turns, then run a private verifier against the final state to produce the reward. Nothing about that sequence is elastic on the timescale of a single rollout. The container is held from the first build second to the last verifier second, whether the agent is thinking, waiting on a model response, or running make.

The reason it deserves its own capacity model is the duty cycle. Across the 238,415 measured rollouts that report token counts, a sandbox is held for a mean of 2,379 seconds while the model emits a mean of 4,085 output tokens over that whole hold, which is 1.72 output tokens per sandbox-second. A fleet sized to keep GPUs busy will be wrong by more than an order of magnitude in both directions at once: too few sandboxes to saturate the decode pool, and far too much decode capacity for the sandboxes that exist.

Why it matters¶

Because the sandbox pool, not the GPU pool, is what a long-horizon agentic RL run runs out of first, and because its cost profile inverts the usual one. The measured campaign spent 191,099 sandbox-hours over 17.0 days to produce 62,511 passing trajectories, which is 3.06 sandbox-hours per usable trajectory. At its 3,000-sandbox peak the whole fleet's aggregate decode demand was only about 5,151 output tokens per second, a load a small serving cluster absorbs without noticing.

It matters a second time because the traffic shape is wrong for the serving stack most teams already have. Agentic terminal rollouts are prefill-dominated by a factor near 28, and in this campaign cached_tokens was zero on all 238,415 rows that report tokens. Every turn re-sent the entire prior transcript. That turns context growth into a quadratic cost in turn count, and it is the single largest lever on the inference bill that costs nothing to pull.

When to size it separately (and when not)¶

Size it separately whenever a rollout allocates a container. Terminal tasks, repository-level SWE tasks, and computer-use tasks all do. The GPU-side ratio in rollout fleet sizing answers a different question: how many generation instances feed one trainer. Neither answer constrains the other.
Size it separately when episodes are long and variable. The measured hold time runs from 3.8 seconds to 26,571, with a median of 1,332 and a p99 of 8,845. A pool sized on the mean will queue badly against that tail.
Do not bother when the environment is in-process. A math verifier, a regex check, or a Python sandbox that runs in milliseconds is a function call, not a fleet. The RLVR verifier patterns cover that case, and this page's arithmetic collapses to nothing.
Do not size against your own average. The single most useful number below is that mean concurrency was 15.6% of peak. Fleets like this are bursty by construction, because a synchronous RL step releases a whole batch of rollouts at once. Capacity is bought at the peak.

Architecture¶

Three measurements decide everything: how long a sandbox is held, what a sandbox costs in node resources, and what token rate one sandbox generates. All three are in the released metadata, and Little's Law connects the first to the fleet size.

# sandbox_fleet.py -- what an agentic terminal-RL rollout fleet costs, measured
# from the 327,189 released RST trajectories and the 37,484 released task.toml
# resource requests, then converted into a node count and a token demand.
# Constants marked MEASURED were computed from the two HuggingFace metadata
# parquets; every other number below is a model whose assumption is stated.
import numpy as np

# MEASURED, whole trajectory corpus (327,189 rows)
N_TRAJ = 327_189
SPAN_S = 1_471_279.1          # first started_at to last finished_at, 17.03 days
SANDBOX_S = 687_957_914.0     # sum(finished_at - started_at)
PEAK_CONC = 3_000             # sweep-line peak of overlapping sandbox intervals
EXC_RATE = 0.2937             # has_exception
N_PASSED = 62_511             # reward == 1
N_REWARDED = 245_554          # rows carrying a reward at all; the rest are null

# MEASURED, the 238,415-row subset that reports token counts
TOK_SANDBOX_S = 567_158_849.0
IN_TOK = 27_258_761_216
OUT_TOK = 973_830_402
TURN_TOKENS = 824.0           # fitted per-turn context increment, R2 = 0.9956
MED_IN_AT = {2: 3207, 5: 11760, 11: 52474, 20: 168366, 50: 1044116}  # median input tokens

# MEASURED, task.toml of 37,459 / 37,484 tasks
CPUS, MEM_MB, DISK_MB = 1, 2048, 6144
# Model: one ordinary CPU worker node, no GPU. Change these for your own SKU.
NODE_CORES, NODE_MEM_MB, NODE_DISK_MB = 96, 384 * 1024, 4 * 1024 * 1024


def mean_concurrency(n, span_s, sandbox_s):
    """Little's Law two ways: arrival rate x holding time, and busy-time / span."""
    return (n / span_s) * (sandbox_s / n), sandbox_s / span_s


def slots_per_node():
    """A sandbox slot is bounded by whichever node resource runs out first."""
    return {
        "cpu": NODE_CORES // CPUS,
        "memory": NODE_MEM_MB // MEM_MB,
        "disk": NODE_DISK_MB // DISK_MB,
    }


def nodes_for(concurrency):
    per = min(slots_per_node().values())
    return int(np.ceil(concurrency / per)), per


def token_demand(concurrency):
    """Fleet-wide token rate implied by holding `concurrency` sandboxes busy."""
    return {
        "prefill_tok_s": IN_TOK / TOK_SANDBOX_S * concurrency,
        "decode_tok_s": OUT_TOK / TOK_SANDBOX_S * concurrency,
    }


def uncached_prefill(turns):
    """Context re-sent every turn: sum of prefix lengths, quadratic in turns."""
    return TURN_TOKENS * turns * (turns + 1) / 2.0


def cache_saving(turns):
    """Ratio of uncached prefill to the final context length alone."""
    return uncached_prefill(turns) / (TURN_TOKENS * turns)


rate_conc, busy_conc = mean_concurrency(N_TRAJ, SPAN_S, SANDBOX_S)

# (1) The two Little's Law routes agree, so the interval arithmetic is sound.
assert np.isclose(rate_conc, busy_conc)

# (2) The fleet was provisioned for 3,000 concurrent sandboxes but averaged under
#     a sixth of that. Peak capacity, not mean load, is what has to be bought.
duty = busy_conc / PEAK_CONC
assert 0.10 < duty < 0.20

# (3) CPU is the binding resource on an ordinary node, not memory or disk.
per = slots_per_node()
assert per["cpu"] < per["memory"] < per["disk"]
n_nodes, slots = nodes_for(PEAK_CONC)

# (4) The rollout is prefill-dominated by a factor near 28, and decode per
#     sandbox-second is under two tokens: the GPU is idle while the box thinks.
d = token_demand(1.0)
assert 27.0 < d["prefill_tok_s"] / d["decode_tok_s"] < 29.0
assert d["decode_tok_s"] < 2.0

# (5) The quadratic model reproduces the measured medians for turn counts of 5 and
#     up, and fails on very short trajectories where fixed prompt overhead dominates.
err = {n: uncached_prefill(n) / v - 1.0 for n, v in MED_IN_AT.items()}
assert all(abs(err[n]) < 0.10 for n in (5, 11, 20, 50))
assert err[2] < -0.20

# (6) Prefix caching would cut prefill by (turns + 1) / 2, and buys nothing at all
#     on a single-turn rollout. cached_tokens is 0 on all 238,415 rows measured.
assert np.isclose(cache_saving(1), 1.0)
assert cache_saving(11) > 5.0 and cache_saving(21) > 10.0

# (7) Cost per useful trajectory: exceptions burn a sandbox and almost never pass.
#     A quarter of rows never got a reward at all, so the honest denominator is the
#     62,511 rollouts that actually passed, not a pass rate applied to every row.
pass_rate = N_PASSED / N_REWARDED
sandbox_h_per_pass = SANDBOX_S / 3600.0 / N_PASSED
assert 0.25 < pass_rate < 0.26 and sandbox_h_per_pass > 3.0

print(f"mean concurrency {busy_conc:.1f} sandboxes, peak {PEAK_CONC}, duty {duty:.1%}")
print(f"slots per node {per}, binding = cpu at {slots}; peak needs {n_nodes} nodes")
print(f"per sandbox-second: prefill {d['prefill_tok_s']:.1f} tok, decode {d['decode_tok_s']:.2f} tok")
peak = token_demand(PEAK_CONC)
print(f"at peak: prefill {peak['prefill_tok_s']:,.0f} tok/s, decode {peak['decode_tok_s']:,.0f} tok/s")
print("quadratic-model error vs measured median input tokens:",
      {n: f"{e:+.1%}" for n, e in err.items()})
print("prefill saving from prefix caching:",
      {n: round(float(cache_saving(n)), 1) for n in (1, 5, 11, 21, 50)})
print(f"exception rate {EXC_RATE:.1%}, pass rate {pass_rate:.1%}, "
      f"{sandbox_h_per_pass:.2f} sandbox-hours per passing trajectory")
print(f"whole campaign: {SANDBOX_S / 3600:,.0f} sandbox-hours over {SPAN_S / 86400:.1f} days")

Executed output:

mean concurrency 467.6 sandboxes, peak 3000, duty 15.6%
slots per node {'cpu': 96, 'memory': 192, 'disk': 682}, binding = cpu at 96; peak needs 32 nodes
per sandbox-second: prefill 48.1 tok, decode 1.72 tok
at peak: prefill 144,186 tok/s, decode 5,151 tok/s
quadratic-model error vs measured median input tokens: {2: '-22.9%', 5: '+5.1%', 11: '+3.6%', 20: '+2.8%', 50: '+0.6%'}
prefill saving from prefix caching: {1: 1.0, 5: 3.0, 11: 6.0, 21: 11.0, 50: 25.5}
exception rate 29.4%, pass rate 25.5%, 3.06 sandbox-hours per passing trajectory
whole campaign: 191,099 sandbox-hours over 17.0 days

The node count is the headline and it is small: 32 ordinary CPU nodes carry 3,000 concurrent sandboxes, because a task asks for one vCPU and 2 GB and an ordinary 96-core node therefore runs out of cores long before memory or disk. That is not a GPU problem, it is a fleet-of-cheap-boxes problem, and it is routinely under-provisioned because it does not appear in any GPU capacity plan.

The duty cycle is the trap. The fleet peaked at 3,000 concurrent sandboxes while averaging 467.6. The peak is a configured cap rather than an observed maximum, and the sweep line proves it: concurrency touches exactly 3,000 on 5,000 separate event intervals, never once exceeds it, and sits at the ceiling for a cumulative 10,248 seconds, 0.70% of the campaign. Buying to the average leaves that cap binding whenever a step releases a batch; buying to the peak leaves 84% of the fleet idle in the mean. Both are correct answers to different questions, and the gap between them is why this fleet wants preemptible or spot capacity rather than reserved.

How to size it¶

The procedure is four steps, and each one needs a measurement you can take on your own fleet in an afternoon.

1. Measure holding time, not episode length. The hold starts at image build and ends when the verifier exits. Turn count is a poor proxy: measured passing trajectories have a median of 9 turns and failing ones 11, yet both hold a sandbox for about the same 1,300 seconds. Take the median and the p99 from your own started_at and finished_at, because the queue is set by the tail.

2. Apply Little's Law to get concurrency. Required concurrency is arrival rate times mean holding time. For a synchronous RL step with a batch of B rollouts that must all finish before the update, the binding constraint is instead B itself, because they start together. That is exactly what produces the 6.4x peak-to-mean ratio measured here.

3. Convert concurrency to nodes through the binding resource. Read the resource requests out of the task bundles rather than guessing: in this pool 37,459 of 37,484 tasks ask for exactly 1 vCPU, 2 GB memory, and 6 GB storage. Divide node capacity by the request along each axis and take the minimum. Disk is the one to check on your own SKU, since 6 GB per slot times a few thousand slots is tens of terabytes of scratch that has to be fast enough to build container layers on.

4. Derive the inference load from concurrency, not from trajectory count. One busy sandbox generates about 48 prefill tokens and 1.7 decode tokens per second. Multiply by the concurrency you sized, not by the number of rollouts you intend to run. At the 3,000-sandbox peak that is 144,186 prefill tokens per second against 5,151 decode.

The prefill problem¶

The 28:1 ratio is not an incidental property of this workload; it is what a multi-turn agent loop does to a serving stack, and the released corpus shows it cleanly enough to model.

Fitting median input tokens against turn count over the 238,415 trajectories that report both gives a quadratic in turn count with R2 = 0.9956, against R2 = 0.8560 for a linear fit. The fitted per-turn context increment is 824 tokens and it is remarkably stable, landing between 767 and 823 across turn counts from 5 to 50. Quadratic growth in turn count is the signature of re-sending the entire transcript on every turn, and the corpus confirms it directly: cached_tokens is 0 on every one of those rows.

The consequence is that prefix caching is worth (turns + 1) / 2 on prefill: 6x at the measured median of 11 turns, 11x at the p90 of 21, and 25.5x on a 50-turn trajectory. Nothing about the training run changes, no reward is affected, and the tokens removed are ones the model already processed. The mechanics are in prompt caching for provider APIs and in KV cache fundamentals plus KV cache inference speedup for a self-hosted serving pool; the RL-specific variant that shares prefixes across a rollout group is in rollout redundancy.

Two cautions on the model. It fails on very short trajectories, overshooting by nothing at 50 turns but undershooting the two-turn median by 22.9%, because a fixed instruction and system prompt dominate before the transcript does. And the saving is an upper bound: it assumes the prefix is byte-identical turn to turn, which a harness breaks the moment it injects a timestamp, reorders tool definitions, or truncates from the front. Verify the cache hit rate rather than assuming it, since the measured campaign shows the failure mode is silent.

How to run it in production¶

Buy peak, on interruptible capacity. A 15.6% duty cycle on reserved instances is a bad trade. Sandbox work is idempotent and individually cheap to lose, which is the ideal spot-instance profile, unlike the trainer pool it feeds.
Cap concurrency explicitly and record the cap. The measured ceiling of 3,000 is hit 5,000 times and never crossed, so it is a configured limit, and it is the number that determines whether the RL step waits. A cap that is not written down becomes an unexplained throughput plateau.
Budget for a large exception rate and gate on it. 29.4% of rollouts carried an exception, and those rollouts passed 0.97% of the time against 26.65% for clean ones. Exceptions are close to a total loss of the sandbox-hours they consume. Alert on the rate per task and per node, because a single bad image or a full disk shows up here first.
Filter the corpus by model before training on it. In the released trajectories the gpt-oss-120b slice ran at a 74.5% exception rate with a 139-second median hold, against 5.9% and 1,023 seconds for Qwen3.6-27B-base. Aggregate statistics over a mixed corpus describe no real configuration.
Expect a quarter of your telemetry to be missing. 81,635 of 327,189 rows (25.0%) carry no reward, 88,774 (27.1%) carry no token or turn counts, and 18,041 (5.5%) have no trajectory attached. Design the accounting so a missing field is visible rather than silently averaged away.
Pin base images and pre-pull them. Every rollout builds from a Dockerfile with a median 600-second build budget. Layer cache hit rate across a fleet is the difference between a build-bound and a run-bound pipeline, and it is the cheapest single intervention after prefix caching.
Separate the synthesis queue from the rollout queue. Generating new tasks (verified task synthesis) reserves the same containers and is throughput-bound, while rollout is latency-bound and stalls a trainer. One shared pool lets the elastic workload starve the inelastic one.

Failure modes¶

Sizing the fleet from GPU utilisation. Decode demand per sandbox is under two tokens a second, so a GPU-derived estimate of how many rollouts are in flight will be wrong by more than an order of magnitude. Size from holding time.
Sizing from the mean when the workload is batched. A synchronous step starts B rollouts at once. Mean concurrency was 15.6% of peak here for exactly that reason, and a pool built to the mean turns every step into a queue.
Silent prefix-cache misses. The measured campaign shows zero cached tokens across 27.3 billion input tokens. Nothing in the pipeline reported an error. Scrape the cache hit counter from the serving pool and alert on it, because the symptom is a bill, not a failure.
Counting a rollout as useful because it completed. Every row in the corpus has status = completed, including the 96,097 with exceptions and the 183,043 with reward 0. Completion is not success; the reward field is, and a quarter of rows do not have one.
Storage exhaustion at the tail. Bundle sizes are median 39.9 kB but reach 16.7 MB, and each slot requests 6 GB of scratch. Provision on the request, not the median bundle, and watch for the trajectory tail: uncompressed trajectory records run to 498 MB at the maximum.
Ignoring hold-time variance in the admission controller. Holds span 3.8 to 26,571 seconds. An admission policy tuned to the median will admit long-running work into a full pool and stall the batch behind it. Admit against the timeout ceiling and reclaim early.
Assuming these constants transfer. They come from one 17-day campaign, one harness (Terminus-2 on Harbor), one sandbox provider (Daytona), and one task pool. The structure transfers: prefill dominance, CPU-bound slots, a bursty duty cycle, and a large exception budget. The numbers should be re-measured.

References¶

Li et al., "Recursive Synthesis for Long-Horizon Terminal Tasks" (RST), arXiv:2608.05466v1, 2026-08-05. https://arxiv.org/abs/2608.05466
Recursive-Task-Synthesis-Trajectories dataset, the 327,189-trajectory corpus measured on this page. https://huggingface.co/datasets/Zhongzhi1228/Recursive-Task-Synthesis-Trajectories
Recursive-Task-Synthesis dataset, source of the task.toml resource requests. https://huggingface.co/datasets/Zhongzhi1228/Recursive-Task-Synthesis
Harbor: a framework for evaluating and optimizing agents and models in container environments (the harness and grading layer). https://doi.org/10.5281/zenodo.20953922
Merrill et al., "Terminal-Bench: Benchmarking agents on hard, realistic tasks in command line interfaces", arXiv:2601.11868. https://arxiv.org/abs/2601.11868
Li et al., "Long-Horizon-Terminal-Bench", arXiv:2607.08964. https://arxiv.org/abs/2607.08964
Daytona sandbox documentation. https://www.daytona.io/docs
vLLM automatic prefix caching documentation. https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html