Markdown

Agent Lightning v1.0: harnessed agentic RL¶

Scope: a framework that RL-trains an agent that already exists, without reimplementing its loop inside the trainer. Agent Lightning v1.0 (Microsoft, arXiv 2608.17528, August 2026) puts an OpenAI-compatible proxy between the harness and the policy, records every model call as a training event, and reassembles those calls into rows for a verl backend. This page covers the component split, the agent-side contract, the call-sequence-to-row aggregation, rollout-level credit assignment, and collocated asynchronous rollout. The RL algorithm itself is GRPO; the loss-masking and multi-turn framing is agentic and tool-use RL; staleness and truncated importance sampling belong to async and disaggregated RL systems and the RL orchestrator control loop; sandbox fleet sizing is agentic rollout sandbox fleet; the definition of "harness" is harness architecture. For the field around it see the RL libraries overview and OpenTinker.

Claims here are checked against the paper PDF (arXiv 2608.17528v1, 18 August 2026, read in full) and against microsoft/agent-lightning at commit e43cbf289e92385e0589e4113fdcfdcb822aebb9 (26 August 2026), cloned and read as source. Every class name, endpoint, and config key below was verified in that tree. Twelve footnotes record source inconsistencies (abstract versus table, paper versus code, paper versus what the repository actually ships) rather than silently resolving them. The YAML, shell, and Python integration snippets are unexecuted reference templates on the real interface. The numpy block is self-contained, was executed, and its output is pasted byte for byte; it models the aggregation and credit-assignment logic, not the GPU training path, and the collocated-async section is a queueing model, not a reproduction of the paper's measurement.

What it is¶

Modern agents run inside a harness that owns context construction, tool execution, and control flow. Early RL frameworks required the agent loop to be reimplemented inside the trainer, which makes reusing mini-SWE-agent, OpenHands, or a bespoke LangChain loop impractical. The original Agent Lightning paper (arXiv 2508.03680, 2025) instead disaggregated the two sides and connected them through an LLM endpoint. The v1.0 paper names the resulting paradigm harnessed agentic RL: RL conducted through the same harness used at deployment, where the harness owns the environment loop and the training system observes only a sequence of request-response pairs across a service boundary.

The paper's formal statement of the difference is the one that drives every design choice downstream. In traditional agentic RL the prompt grows monotonically, p_t = (p_{t-1}, a_{t-1}, o_t) (Equation 1), so a rollout is one linear token trajectory and one training sample. In harnessed agentic RL the training engine records only

C(rho) = ((p_1, a_1), (p_2, a_2), ..., (p_T, a_T))      Equation 2

with the harness and environment state latent between calls, and with no token-prefix relation assumed between consecutive prompts. Assembling that call list into training rows is a modelling decision, and the paper identifies four consequences: retokenisation and sample merging, advantage calculation, loss normalisation, and training-backend scheduling under a dynamic row count.

Three components implement this. The API Gateway (agl-server, a FastAPI app) stores rollouts, registered model endpoints, and append-only events, and forwards agent model calls to the inference servers the trainer registered. The Rollout Controller (agl-controller) reconciles queued rollouts into Kubernetes Jobs or local subprocesses. The Customized Trainer subclasses verl's RayPPOTrainer as AgentLightningRayPPOTrainer, registers rollouts, waits for them to reach a terminal state, pulls their events, and assembles training rows.

v1.0 is a complete rewrite, not an increment. The v0.x line (last release 0.3.0; 0.3.1 is only the in-development version string on the v0x branch) carried LightningStore with in-memory, SQLite, and MongoDB backends, LitAgent, an OpenTelemetry tracer stack with AgentOps, LiteLLM, vLLM, and Weave instrumentation, an adapter and algorithm plugin system, and 30,563 lines across 93 Python files under agentlightning/. v1.0.1 has 27 Python files and 4,758 lines,¹ agentlightning/__init__.py exports nothing but __version__, and there is no Python-level agent API left: the contract is HTTP plus three environment variables.

Why use it¶

The deployment harness is the training harness. Context policy, tool protocols, retry behaviour, and prompt templates stay exactly as they ship, which removes the train-serve gap that reimplementing the loop inside verl reintroduces.
Integration is an endpoint change. An OpenAI-compatible client pointed at AGL_OPENAI_BASE_URL plus one HTTP POST of a reward is the whole agent-side contract. No import of agentlightning is required in the agent process.
Agent execution runs on Kubernetes you already own. verl Uni-Agent uses Modal Sandbox or Volcano veFaaS and slime uses E2B; Agent Lightning schedules each rollout as a standard Kubernetes Job from a Jinja template, so the whole stack stays self-hosted. Fleet sizing is the same arithmetic as any agentic rollout sandbox fleet.
The correctness choices are explicit and configurable. Rollout-level advantage and rollout-level loss normalisation are named settings with a stated rationale. The two settings do not sort the field the same way: on advantage the paper places verl Uni-Agent and Polar at the rollout level and slime and AReaL at the sample level, while on loss normalisation it names only slime, and names it on the same side as Agent Lightning.²
A complete coding-agent recipe ships with it. Data cleaning, reward-hacking safeguards, Kubernetes templates, and training scripts for SWE-smith, on four GPUs rather than a large cluster.

When to use it (and when not)¶

Use it when the agent already exists and is worth preserving, when the harness is multi-agent or spawns subagents or summarises context (all of which break the single-linear-trajectory assumption), when rollouts are long and heavy-tailed enough that synchronous RL wastes the GPU pool, and when the rollout environment must be self-hosted for cost or data-residency reasons.

Do not use it when the agent is a single-turn completion, where plain GRPO on verl is simpler and faster. Do not use it when the harness streams responses: the proxy rejects stream: true with HTTP 400.¹² Do not use it when the inference backend cannot return token IDs and log probabilities through the OpenAI API, since the trainer trains on exact sampled token IDs, never on retokenised text. Do not reach for it to get better within-trajectory credit assignment: every training row of a rollout inherits the same scalar outcome reward, and the paper states plainly that "Future work may still be needed to design better credit assignment across the samples within a rollout." For turn-level credit see TRACE.

Architecture¶

flowchart TB
  subgraph AGENT["Agent execution cluster (Machine A)"]
    CTRL["Rollout Controller (agl-controller)<br/>K8sReconciler or LocalReconciler<br/>poll_interval 5s, max_jobs_per_minute 100"]
    JOB["One Kubernetes Job per rollout<br/>backoffLimit 0, activeDeadlineSeconds<br/>harness runs unmodified"]
  end
  subgraph GW["API Gateway (agl-server, FastAPI, port 8080)"]
    STORE["In-memory store<br/>rollouts, models, events"]
    PROXY["OpenAI proxy<br/>/proxy/rollout/{id}/attempt/{aid}/mode/{mode}/openai/v1"]
  end
  subgraph TRAIN["Training cluster (Machine B)"]
    TR["AgentLightningRayPPOTrainer (verl)<br/>RolloutAdapter, rollout-level advantage<br/>per_rollout_mean loss"]
    VLLM["vLLM replicas serving the current policy<br/>return_token_ids, logprobs"]
  end
  CTRL -->|"GET /api/rollouts?state_in=queuing"| STORE
  CTRL -->|"create Job, PATCH status"| JOB
  JOB -->|"chat/completions"| PROXY
  JOB -->|"POST reward event"| STORE
  PROXY -->|"forward"| VLLM
  PROXY -->|"record model_request event"| STORE
  TR -->|"POST /api/rollouts, POST /api/models"| STORE
  TR -->|"GET events?format=triplet"| STORE
  TR -->|"pause, drain, update weights, resume"| PROXY

What crosses each boundary¶

The Gateway stores three object types (paper Figure 11). A rollout is one agent execution with an id, an input derived from a training example, a RolloutConfig, metadata, and a status. The state machine in agentlightning/schemas.py is QUEUING -> {RUNNING, FAILED} and RUNNING -> {SUCCEEDED, FAILED}, with both terminal states final and transitions enforced by the store (an invalid transition returns HTTP 409). A model is a (name, endpoint, version) triple the trainer registers. An event attaches arbitrary data to a rollout; two types are well known, model_request (prompt token ids, response token ids, chosen-token log probabilities, latency, HTTP status, retry count) and reward (a scalar value).

Rollouts are not one-to-one with training examples: GRPO creates actor_rollout_ref.rollout.n independent rollouts per example, each with its own id, all sharing one data_id that becomes the group key.

Every Gateway rollout endpoint is idempotent, so the trainer and Controller can retry freely after a network failure: rollout creation with a caller-supplied rollout_id returns the existing rollout unchanged, model registration upserts by (model, endpoint), and deletion of a missing id is a no-op. A retried model call cannot be made idempotent the same way, because each retry is a fresh generation, so duplicates are removed at read time: GET /api/rollouts/{id}/events?format=triplet keeps only the last model_request for each distinct prompt_token_ids key and discards the superseded ones.⁹

The proxy path embeds the rollout id, so every model call attributes itself. Paper Table 1 lists nine endpoints; the code at this commit serves fifteen, the extras being cursor pagination over terminal rollouts, rollout deletion, the pause/resume/state management routes that collocated async depends on, and /healthz.⁸

What the agent must expose, and what it does not change¶

The Controller injects exactly three environment variables into each Job container or subprocess (agentlightning/controller/k8s_reconciler.py, local_reconciler.py):

Variable	Value
`AGL_OPENAI_BASE_URL`	`{gateway}/proxy/rollout/{rollout_id}/attempt/0/mode/{train\\|val}/openai/v1`
`AGL_EVENT_URL`	`{gateway}/api/rollouts/{rollout_id}/attempt/0/events`
`AGL_KEY`	shared bearer key

The agent points its OpenAI client at the first, POSTs {"event_type": "reward", "data": {"value": <float>}} to the second at the end of the episode, and authenticates with the third. Everything else, tools, memory, subagents, prompt templates, retries, is untouched. What the framework silently changes on the way through is narrower than "zero changes" implies: ProxyRouter.prepare_body overwrites model with the server's configured default_proxy.model_name, overwrites temperature with default_proxy.train.temperature (default 1) or default_proxy.val.temperature (default 0.7), and forces return_token_ids: true on every proxied request. logprobs: true is narrower: prepare_body sets it only on the train-mode branch and only when default_proxy.include_log_probs is true (proxy.py:66-72; the key defaults to true in agentlightning/config/server.yaml), so val-mode calls never get it and a Gateway configured with include_log_probs: false never gets it at all. A harness that relies on temperature=0 for a deterministic sub-step will not get it during training.¹²

From a call sequence to training rows¶

RolloutAdapter (agentlightning/verl/rollout_adapter.py) implements two aggregation levels, selected by agentlightning.trace_aggregator.level.

transition emits one row per model call and merges nothing. It is token-exact by construction and recomputes every shared prefix, which is the expensive option the paper names first.

trajectory, the shipped default, is the paper's "best-effort sequence merging". Call i+1 is merged into the current row only when ids_startswith(prompt_ids, current_context) holds at the token level, where current_context is the previous prompt plus the previous sampled response. Tokens the harness inserted between the two calls (tool observations, system reminders) are appended to the response with response_mask = 0, so they are context for attention but carry no gradient, exactly the masking discipline described in chat rendering and loss masking. When the prefix test fails, the current row is closed and a new one starts, and the mismatch is logged to a W&B table with the decoded previous and current traces.

The paper's contribution here is naming why the prefix test fails even when the text is identical: chat templates are not compositional (Template(A || B) != Template(A) || Template(B), Equation 10, with Qwen's template stripping an earlier <think> marker as the worked example), decode then retokenise is not the identity (Tok(Decode(a)) != a, Equation 11, with "having" sampled as h + aving and retokenised as hav + ing), and tool-call parsers reserialise structured output before it re-enters the next prompt. AReaL and verl Uni-Agent buffer the original token ids in the proxy and splice them back into the next prompt; the paper argues this makes training off-policy, because the response was sampled under the real prompt p_{i+1}, not the stitched one (Equations 12 and 13). Agent Lightning deliberately keeps no server-side request buffer and accepts a lower merge ratio instead.

The measured cost of that choice, on the paper's own coding-agent run (Figure 10): only 36% of rollouts stay a single training row, and a rollout yields 2.41 rows on average.

Credit assignment and normalisation, validated¶

Once a rollout can produce N_rho rows, two questions have answers that differ across frameworks. Should the GRPO baseline be the mean over rollouts or over rows? And should the loss be normalised per row or per rollout? Agent Lightning answers "rollout" to both, on the grounds that the row count is driven by incidental factors. The framework attributions differ between the two questions and are easy to collapse. For advantage, Section 2.2 reports that verl Uni-Agent and Polar compute at the rollout level while slime and AReaL compute at the sample level. For loss normalisation, Section 2.3 names one framework only, and it is slime: Equation 16, the rollout-level token-mean form that Agent Lightning ends up preferring, is slime's implementation, not a departure from it.²

The block below models the whole path in numpy: the trajectory merge, the Figure 4 baseline split, what rollout-level advantage does not fix, the three loss normalisations of Equations 14 to 16 on the Figure 5 batch, the mini-batch alignment drop, the failure path for errored and retried calls, and a queueing model of synchronous versus collocated-asynchronous stepping. Every section asserts, including a rejected-input case, a zero-variance boundary, and a crashed-rollout failure case. Run: python3 agent_lightning_credit.py.

# agent_lightning_credit.py -- Agent Lightning v1.0 transition construction, rollout-level
# credit assignment, and collocated-async arithmetic. Runnable, numpy only.
#
# Ports, by hand, from microsoft/agent-lightning @ e43cbf289e92385e0589e4113fdcfdcb822aebb9:
#   agentlightning/verl/rollout_adapter.py         trajectory / transition aggregation
#   agentlightning/verl/rollout_level_advantage.py rollout-level advantage broadcast
#   agentlightning/verl/per_rollout_loss.py        rollout-level token-mean normalisation
#   agentlightning/verl/trainer.py                 zero-variance-group drop preference
#   agentlightning/server/routes/events.py         prompt dedupe of retried model calls
# plus Equations 14/15/16 and Figures 4/5 of arXiv:2608.17528.

import numpy as np

rng = np.random.default_rng(20260826)

# ---------------------------------------------------------------------------
# 1) Trace aggregation. Port of RolloutAdapter.get_train_data_batch.
# ---------------------------------------------------------------------------


def merge_trajectory(calls):
    """level='trajectory': merge call i+1 into call i only when the new prompt is an
    EXACT token-level prefix extension of (prompt_i, response_i). Harness-inserted
    observation tokens enter the response with mask 0: context, no gradient."""
    prompt, resp = list(calls[0][0]), list(calls[0][1])
    mask, ctx = [1] * len(resp), list(calls[0][0]) + list(calls[0][1])
    rows = []
    for p, r in calls[1:]:
        nxt = list(p) + list(r)
        if list(p[: len(ctx)]) == ctx:                        # ids_startswith(prompt_ids, current_context)
            if len(p) > len(ctx):
                obs = list(p[len(ctx):])
                resp += obs
                mask += [0] * len(obs)
            resp += list(r)
            mask += [1] * len(r)
            ctx = nxt
            continue
        rows.append((prompt, resp, mask))                     # prefix broken: close row, start a new one
        prompt, resp, mask, ctx = list(p), list(r), [1] * len(r), nxt
    rows.append((prompt, resp, mask))
    return rows


def transition_rows(calls):
    """level='transition': one training row per model call, nothing merged."""
    return [(list(p), list(r), [1] * len(r)) for p, r in calls]


def build_react_calls(n_turns, retoken_break_at=None, seed=0):
    """A ReAct harness: prompt_{i+1} = prompt_i + action_i + observation_i."""
    g = np.random.default_rng(seed)
    tok = iter(range(1000, 100000))
    prompt, prev_len = [next(tok) for _ in range(12)], 0
    calls, actions = [], []
    for i in range(n_turns):
        p = list(prompt)
        if retoken_break_at is not None and i == retoken_break_at:
            # Decode-retokenize drift inside the PREVIOUS action: one sampled token
            # ("h","aving") comes back re-split ("hav","ing"). Same text, new ids.
            p = p[:prev_len] + [999_999, 999_998] + p[prev_len + 1:]
        action = [next(tok) for _ in range(int(g.integers(4, 9)))]
        obs = [next(tok) for _ in range(int(g.integers(3, 7)))]
        calls.append((p, action))
        actions.append(action)
        prev_len, prompt = len(p), p + action + obs
    return calls, actions


calls, actions = build_react_calls(6, seed=1)
n_action_tokens = sum(len(a) for a in actions)
rows = merge_trajectory(calls)
trows = transition_rows(calls)

assert len(rows) == 1, f"clean ReAct trace must merge to one row, got {len(rows)}"
assert len(trows) == 6
# Invariant: aggregation never creates or destroys a trained token.
assert sum(sum(m) for _, _, m in rows) == n_action_tokens
assert sum(sum(m) for _, _, m in trows) == n_action_tokens
# Trajectory mode carries observation tokens as masked context; transition mode does not.
ctx_tokens = sum(len(m) - sum(m) for _, _, m in rows)
assert ctx_tokens > 0 and sum(len(m) - sum(m) for _, _, m in trows) == 0

bcalls, bactions = build_react_calls(6, retoken_break_at=3, seed=1)
brows = merge_trajectory(bcalls)
assert len(brows) == 2, f"one retokenisation break must give exactly 2 rows, got {len(brows)}"
assert sum(sum(m) for _, _, m in brows) == sum(len(a) for a in bactions), "break lost trained tokens"
print(f"[1] clean 6-turn trace: {len(rows)} merged row ({ctx_tokens} masked observation tokens) "
      f"vs {len(trows)} transition rows; {n_action_tokens} trained tokens either way")
print(f"    one retokenisation break at turn 3 -> {len(brows)} rows, tokens conserved")

# ---------------------------------------------------------------------------
# 2) Advantage baseline: rollout level vs sample level (paper Figure 4).
# ---------------------------------------------------------------------------


def grpo_advantage(rewards, norm_by_std=True, ddof=1):
    r = np.asarray(rewards, dtype=np.float64)
    a = r - r.mean()
    return a / (r.std(ddof=ddof) + 1e-6) if norm_by_std else a


rollout_rewards = np.array([1.0, 0.0])            # rollout 1 -> 3 rows, rollout 2 -> 1 row
sample_rewards = np.array([1.0, 1.0, 1.0, 0.0])
assert rollout_rewards.mean() == 0.5, "rollout-level baseline must be 1/2"
assert sample_rewards.mean() == 0.75, "sample-level baseline must be 3/4"

a_roll = grpo_advantage(rollout_rewards, norm_by_std=False)
a_samp = grpo_advantage(sample_rewards, norm_by_std=False)
assert np.allclose(a_roll, [0.5, -0.5])
assert np.allclose(a_samp, [0.25, 0.25, 0.25, -0.75])
# Retokenisation alone halves the winner's advantage and inflates the loser's by 1.5x.
assert np.isclose(a_samp[0] / a_roll[0], 0.5)
assert np.isclose(a_samp[-1] / a_roll[1], 1.5)


def compute_rollout_level_advantage(row_rollout_ids, row_rewards, row_uids, norm_by_std=True):
    """Port of compute_rollout_level_advantage: one representative row per rollout_id,
    advantage computed there, broadcast to every row of that rollout."""
    first = {}
    for i, rid in enumerate(row_rollout_ids):
        first.setdefault(rid, i)
    for rid, i0 in first.items():                              # _validate_same_reward / _validate_same_uid
        idx = [i for i, x in enumerate(row_rollout_ids) if x == rid]
        assert len({row_rewards[i] for i in idx}) == 1, f"rollout {rid}: non-constant reward"
        assert len({row_uids[i] for i in idx}) == 1, f"rollout {rid}: multiple uid values"
    by_uid = {}
    for rid, i0 in first.items():
        by_uid.setdefault(row_uids[i0], []).append(rid)
    scalar = {}
    for rids in by_uid.values():
        for rid, a in zip(rids, grpo_advantage([row_rewards[first[r]] for r in rids], norm_by_std)):
            scalar[rid] = a
    return np.array([scalar[rid] for rid in row_rollout_ids])


rid_rows, rew_rows, uid_rows = ["r1", "r1", "r1", "r2"], [1.0, 1.0, 1.0, 0.0], ["g0"] * 4
bcast = compute_rollout_level_advantage(rid_rows, rew_rows, uid_rows, norm_by_std=False)
assert np.allclose(bcast, [0.5, 0.5, 0.5, -0.5]), bcast
slow = dict(zip(["r1", "r2"], grpo_advantage([1.0, 0.0], norm_by_std=False)))   # slow reference
assert np.allclose(bcast, [slow[r] for r in rid_rows])

# Failure case: rows of one rollout disagreeing on the reward are rejected, not averaged.
try:
    compute_rollout_level_advantage(["r1", "r1"], [1.0, 0.0], ["g0", "g0"])
    raise SystemExit("expected non-constant reward to raise")
except AssertionError as exc:
    assert "non-constant reward" in str(exc)

# Boundary: a zero-variance group contributes exactly zero gradient.
assert np.allclose(grpo_advantage([1.0, 1.0, 1.0, 1.0]), 0.0)
print(f"[2] Figure 4 baselines: rollout-level {rollout_rewards.mean():.2f}, "
      f"sample-level {sample_rewards.mean():.2f}")
print(f"    winner advantage {a_roll[0]:+.2f} -> {a_samp[0]:+.2f} (x0.5), "
      f"loser {a_roll[1]:+.2f} -> {a_samp[-1]:+.2f} (x1.5), from retokenisation alone")

# ---------------------------------------------------------------------------
# 3) What rollout-level advantage does NOT fix: within-rollout credit.
# ---------------------------------------------------------------------------

step_lens = np.array([40.0, 12.0, 90.0, 18.0])                # response tokens per call
causal, A = 2, 0.5                                            # exactly one call caused the outcome
framework = np.full(4, A)                                     # every row inherits the same scalar
share = (framework * step_lens) / (framework * step_lens).sum()
assert np.isclose(share.sum(), 1.0)
assert np.isclose(share[causal], step_lens[causal] / step_lens.sum())
assert np.isclose(share[causal], 0.5625), share[causal]
oracle = np.zeros(4); oracle[causal] = A
oracle_share = (oracle * step_lens) / (oracle * step_lens).sum()
assert np.isclose(oracle_share[causal], 1.0)
assert np.isclose(1.0 - share[causal], 0.4375)
print(f"[3] 1 causal call of 4: gradient share on the causal call {share[causal]:.1%} "
      f"(token-weighted), {1 - share[causal]:.1%} smeared onto non-causal calls")

# ---------------------------------------------------------------------------
# 4) Loss normalisation: Equations 14, 15, 16 on the Figure 5 batch.
# ---------------------------------------------------------------------------

batch = {"A": [50, 100], "B": [30, 30, 30], "C": [40]}        # rollout -> response lengths
T = {rho: float(sum(v)) for rho, v in batch.items()}
N = {rho: len(v) for rho, v in batch.items()}
T_tot, N_tot, R = sum(T.values()), sum(N.values()), len(batch)


def losses(c):
    """c[rho] = per-token loss, constant within rollout rho. Returns Eq 14, 15, 16."""
    eq14 = sum(c[rho] * T[rho] for rho in batch) / T_tot
    eq15 = sum(c[rho] * N[rho] for rho in batch) / N_tot
    eq16 = sum(c[rho] for rho in batch) / R
    return eq14, eq15, eq16


ones = {rho: 1.0 for rho in batch}
assert np.allclose(losses(ones), (1.0, 1.0, 1.0))             # all three agree on a uniform batch


def numeric_weight(scheme_idx, rho, h=1e-6):
    up, dn = dict(ones), dict(ones)
    up[rho] += h; dn[rho] -= h
    return (losses(up)[scheme_idx] - losses(dn)[scheme_idx]) / (2 * h)


for rho in batch:                                             # analytic == finite difference
    assert np.isclose(numeric_weight(0, rho), T[rho] / T_tot)
    assert np.isclose(numeric_weight(1, rho), N[rho] / N_tot)
    assert np.isclose(numeric_weight(2, rho), 1.0 / R)

# Eq 15 (GRPO seq-mean) over-weights rollout B 3x purely because it produced 3 rows.
assert np.isclose(N["B"] / N_tot, 0.5) and np.isclose(N["C"] / N_tot, 1 / 6)
assert np.isclose((N["B"] / N_tot) / (N["C"] / N_tot), 3.0)
# Eq 14 (DAPO token-mean) weights by length: rollout A takes 53.6% of the gradient.
assert np.isclose(T["A"] / T_tot, 150 / 280)
# Eq 16 (Agent Lightning) gives every rollout exactly 1/3.
assert len({round(1.0 / R, 12)}) == 1
print(f"[4] rollout gradient share  Eq14 token-mean A/B/C = "
      f"{T['A']/T_tot:.3f}/{T['B']/T_tot:.3f}/{T['C']/T_tot:.3f}")
print(f"                            Eq15 seq-mean    A/B/C = "
      f"{N['A']/N_tot:.3f}/{N['B']/N_tot:.3f}/{N['C']/N_tot:.3f}")
print(f"                            Eq16 rollout     A/B/C = "
      f"{1/R:.3f}/{1/R:.3f}/{1/R:.3f}")

# What normalize_advantages_by_rollout() actually divides by: the rollout's token count
# TIMES num_trained_rows, not times the number of distinct rollouts.
n_rows = int(N_tot)
code_loss = sum(L / (T[rho] * n_rows) for rho, lens in batch.items() for L in lens)
assert np.isclose(code_loss, losses(ones)[2] * R / n_rows)
assert np.isclose(code_loss * n_rows / R, 1.0)
assert np.isclose(n_rows / R, 2.0)                            # mean rows per rollout in this batch
print(f"[4] per_rollout_loss divides by rollout_tokens * num_trained_rows({n_rows}), not "
      f"* n_rollouts({R}): loss = Eq16 * {R/n_rows:.2f}")

# ---------------------------------------------------------------------------
# 5) Mini-batch alignment drop, zero-variance groups first (trainer.py).
# ---------------------------------------------------------------------------


def align_drop(uids, rewards, mini_bs, max_updates=None):
    n = len(uids)
    keep_n = n // mini_bs * mini_bs
    if max_updates is not None:
        keep_n = min(keep_n, mini_bs * max_updates)
    to_drop = n - keep_n
    same = []
    for uid in dict.fromkeys(uids):
        idx = [i for i, u in enumerate(uids) if u == uid]
        if max(rewards[i] for i in idx) - min(rewards[i] for i in idx) == 0.0:
            same.extend(idx)
    drop = set(same[:to_drop])
    if len(drop) < to_drop:
        drop |= set([i for i in range(n) if i not in drop][: to_drop - len(drop)])
    return [i for i in range(n) if i not in drop]


uids = ["g0"] * 4 + ["g1"] * 4 + ["g2"] * 4                   # g0 is all-1: zero variance
rewards = [1.0] * 4 + [1.0, 0.0, 1.0, 0.0] + [0.0, 1.0, 0.0, 1.0]
kept = align_drop(uids, rewards, mini_bs=8)
assert len(kept) == 8 and all(uids[i] != "g0" for i in kept), "zero-variance rows must go first"
assert len(align_drop(uids, rewards, mini_bs=4, max_updates=1)) == 4     # max_ppo_update_times cap
# Edge: every group zero-variance -> the whole step is gradient-free but still runs.
flat = ["h0"] * 4
assert np.allclose(grpo_advantage([1.0] * 4), 0.0) and len(align_drop(flat, [1.0] * 4, 4)) == 4
print(f"[5] 12 rows, mini_bs=8 -> {len(kept)} kept, all 4 zero-variance rows dropped first")

# ---------------------------------------------------------------------------
# 6) Failure handling in the transition builder.
# ---------------------------------------------------------------------------


def dedupe_by_prompt(events):
    """_dedupe_model_requests_by_prompt_token_ids: keep the LAST call per prompt."""
    last = {}
    for i, e in enumerate(events):
        last[tuple(e["prompt_token_ids"])] = i
    return [e for i, e in enumerate(events) if i in set(last.values())]


def build_triplets(events):
    """_build_completed_rollout: drop errored calls and calls with no response tokens."""
    return [e for e in events
            if e.get("status") != "error"
            and not (isinstance(e.get("http_status"), int) and e["http_status"] >= 400)
            and e.get("response_token_ids")]


evts = [
    {"prompt_token_ids": [1, 2], "response_token_ids": [7], "http_status": 200, "status": "ok"},
    {"prompt_token_ids": [1, 2], "response_token_ids": [8], "http_status": 200, "status": "ok"},
    {"prompt_token_ids": [1, 2, 8, 9], "response_token_ids": [], "http_status": 503, "status": "error"},
    {"prompt_token_ids": [1, 2, 8, 9], "response_token_ids": [11], "http_status": 200, "status": "ok"},
]
deduped = dedupe_by_prompt(evts)
assert len(deduped) == 2 and deduped[0]["response_token_ids"] == [8], "dedupe keeps the last call"
kept_t = build_triplets(deduped)
assert len(kept_t) == 2
# A rollout whose only call errored yields no triplets, hence no training rows at all.
assert build_triplets([evts[2]]) == []
# reward_fillna_value makes a crashed rollout numerically identical to an honest failure.
REWARD_FILLNA = 0.0
assert REWARD_FILLNA == 0.0
crashed, honest_fail = REWARD_FILLNA, 0.0
assert crashed == honest_fail, "crash and genuine task failure are indistinguishable at 0.0"
# ... and a group where every rollout crashed looks like a zero-variance group, so it is
# dropped silently rather than raised.
assert np.allclose(grpo_advantage([REWARD_FILLNA] * 4), 0.0)
print(f"[6] 4 raw model_request events -> {len(deduped)} after prompt dedupe -> "
      f"{len(kept_t)} triplets; all-crashed group advantage = 0 (dropped, not raised)")

# ---------------------------------------------------------------------------
# 7) Sync vs collocated-async step time, heavy-tailed rollout durations.
#    Sizing from examples/swe_smith/train_smith_agent.py:
#    data.train_batch_size=16, actor_rollout_ref.rollout.n=8, async_train_batch_size=50,
#    agentlightning.rollout_timeout_seconds=5400.
# ---------------------------------------------------------------------------

B_TRAIN, GROUP_N, B_ASYNC, STEPS = 16, 8, 50, 120
SIGMA, MEAN_LOG, TIMEOUT = 1.1, np.log(300.0), 5400.0


def dur(shape):
    return rng.lognormal(mean=MEAN_LOG, sigma=SIGMA, size=shape)


def group_time(timeout):
    """A GRPO group completes only when all rollout.n siblings reach a terminal state.
    A sibling that exceeds rollout_timeout_seconds terminates as FAILED at the timeout."""
    return float(min(dur(GROUP_N).max(), timeout))


single_mean = float(dur(400_000).mean())
group_mean = float(np.mean([group_time(np.inf) for _ in range(40_000)]))
sync_mean = float(np.mean([dur((B_TRAIN, GROUP_N)).max() for _ in range(STEPS)]))
assert group_mean > 2.0 * single_mean, "a group of 8 must amplify the tail"
assert sync_mean > 3.0 * group_mean, "the batch max must amplify it again"


def simulate(timeout):
    total = np.array([group_time(timeout) for _ in range(B_ASYNC)])
    remaining, age = total.copy(), np.zeros(B_ASYNC, dtype=int)
    steps, ages_used, consumed_total, offered_total = [], [], [], list(total)
    for _ in range(STEPS):
        consumed = np.argsort(remaining)[:B_TRAIN]
        step_time = float(remaining[consumed].max())   # B_TRAIN-th order statistic, not the max
        steps.append(step_time)
        ages_used.extend((age[consumed] + 1).tolist())
        consumed_total.extend(total[consumed].tolist())
        remaining = np.maximum(remaining - step_time, 0.0)
        fresh = np.array([group_time(timeout) for _ in range(B_TRAIN)])
        total[consumed], remaining[consumed] = fresh, fresh
        offered_total.extend(fresh.tolist())
        age += 1
        age[consumed] = 0
    return float(np.mean(steps[20:])), ages_used, float(np.mean(consumed_total)), float(np.mean(offered_total))


async_mean, ages, consumed_mean, offered_mean = simulate(TIMEOUT)
speedup = sync_mean / async_mean
assert speedup > 1.5, f"collocated async should beat sync on a heavy tail, got {speedup:.2f}x"
mean_age, max_age = float(np.mean(ages)), int(max(ages))
assert max_age >= 2, "async must produce cross-policy-version rollouts"
assert mean_age > 1.0
# The timeout is the only staleness ceiling: a group cannot survive more optimizer
# steps than fit inside rollout_timeout_seconds.
ceiling = int(np.ceil(TIMEOUT / async_mean)) + 1
assert max_age <= ceiling, (max_age, ceiling)
# Without a timeout, nothing evicts a slow group and carry-over age runs far higher.
_, ages_untimed, _, _ = simulate(np.inf)
assert max(ages_untimed) > max_age, (max(ages_untimed), max_age)
# Taking the first B_TRAIN groups to finish looks like it should length-bias every
# batch toward short trajectories. Carry-over cancels it: a slow group is delayed,
# never skipped, so in steady state the consumed distribution matches the offered one.
# The cost of collocated async is staleness, not selection bias.
bias = abs(1.0 - consumed_mean / offered_mean)
assert bias < 0.02, bias
timeout_rate = float(np.mean([group_time(TIMEOUT) >= TIMEOUT for _ in range(20_000)]))
assert timeout_rate > 0.0, "some groups must hit the timeout at this tail"
print(f"[7] mean rollout {single_mean:6.1f}s -> group-of-{GROUP_N} max {group_mean:7.1f}s "
      f"({group_mean/single_mean:.2f}x) -> sync step {sync_mean:7.1f}s ({sync_mean/single_mean:.1f}x)")
print(f"    collocated-async step {async_mean:7.1f}s, speedup {speedup:.2f}x, "
      f"{B_ASYNC - B_TRAIN} of {B_ASYNC} groups carried over each step")
print(f"    staleness at consumption: mean {mean_age:.2f} steps, max {max_age} "
      f"(timeout ceiling {ceiling}); without a timeout max reaches {max(ages_untimed)}")
print(f"    groups hitting rollout_timeout_seconds={TIMEOUT:.0f}: {timeout_rate:.1%}")
print(f"    length bias: consumed groups average {consumed_mean:.1f}s vs {offered_mean:.1f}s offered "
      f"({bias:.2%} apart, carry-over cancels the selection bias)")

d = dur((4000, B_TRAIN * GROUP_N))
idle = float((1.0 - d.mean(axis=1) / d.max(axis=1)).mean())
assert idle > 0.5, idle
print(f"[7] sync agent-worker idle fraction waiting on the straggler: {idle:.1%}")

print("\nall assertions passed")

Executed output:

[1] clean 6-turn trace: 1 merged row (24 masked observation tokens) vs 6 transition rows; 38 trained tokens either way
    one retokenisation break at turn 3 -> 2 rows, tokens conserved
[2] Figure 4 baselines: rollout-level 0.50, sample-level 0.75
    winner advantage +0.50 -> +0.25 (x0.5), loser -0.50 -> -0.75 (x1.5), from retokenisation alone
[3] 1 causal call of 4: gradient share on the causal call 56.2% (token-weighted), 43.8% smeared onto non-causal calls
[4] rollout gradient share  Eq14 token-mean A/B/C = 0.536/0.321/0.143
                            Eq15 seq-mean    A/B/C = 0.333/0.500/0.167
                            Eq16 rollout     A/B/C = 0.333/0.333/0.333
[4] per_rollout_loss divides by rollout_tokens * num_trained_rows(6), not * n_rollouts(3): loss = Eq16 * 0.50
[5] 12 rows, mini_bs=8 -> 8 kept, all 4 zero-variance rows dropped first
[6] 4 raw model_request events -> 2 after prompt dedupe -> 2 triplets; all-crashed group advantage = 0 (dropped, not raised)
[7] mean rollout  549.1s -> group-of-8 max  1842.3s (3.35x) -> sync step  6012.1s (10.9x)
    collocated-async step   672.7s, speedup 8.94x, 34 of 50 groups carried over each step
    staleness at consumption: mean 3.09 steps, max 9 (timeout ceiling 10); without a timeout max reaches 26
    groups hitting rollout_timeout_seconds=5400: 3.5%
    length bias: consumed groups average 1752.2s vs 1763.0s offered (0.61% apart, carry-over cancels the selection bias)
[7] sync agent-worker idle fraction waiting on the straggler: 88.8%

all assertions passed

Four results carry over to configuration decisions. Section 2 shows the sample-level baseline is not a rounding difference: on the paper's own Figure 4 batch it halves the winning rollout's advantage and inflates the losing one by 1.5x, purely because retokenisation split the winner. Section 3 shows the limit of the fix: with one causal call among four, the causal call still receives only its token-weighted share of the gradient (56.2% here), so 43.8% lands on calls that did not cause the outcome. Section 4 shows Equation 15, the GRPO default, giving a three-row rollout three times the weight of a one-row rollout, and also shows that normalize_advantages_by_rollout divides by num_trained_rows rather than the number of distinct rollouts, which preserves equal relative weighting but leaves a batch-dependent global scale factor.¹⁰ Section 7 shows why synchronous stepping is untenable for long agents: a group of eight amplifies the mean rollout duration 3.35x, the batch maximum amplifies it again to 10.9x, and the agent workers sit idle 88.8% of the time waiting on the straggler.

How to use it¶

Install pins the training stack explicitly. scripts/setup_verl.sh accepts verl 0.7.1 (with vLLM 0.12.0) or 0.8.0 (with vLLM 0.20.2), and CUDA wheel variant cu129 or cu130. The upper bound verl<0.9.0 in the verl-cpu group of pyproject.toml is deliberate: verl 0.9.0 renamed main_ppo.TaskRunner and dropped create_rl_sampler, both of which entrypoint.py imports.

Reference template, not executed here; verify against the installed versions:

# agent-lightning 1.0.1, verl 0.8.0, vLLM 0.20.2, CUDA 13.0
uv sync
bash scripts/setup_verl.sh 0.8.0 cu130

# Machine B (GPUs): gateway first, it is the source of truth for rollout state
agl-server host=0.0.0.0 port=8080 key="$AGL_KEY" \
  default_proxy.model_name=Qwen/Qwen3.5-9B

# Machine A (cluster access): controller reconciles rollouts into Jobs
agl-controller runner_type=k8s \
  agl_server.url="http://$GATEWAY_HOST:8080" \
  agl_server.key="$AGL_KEY" \
  k8s_runner.namespace=agents k8s_runner.ttl_after_finished=600

# Machine B: trainer registers models, enqueues rollouts, collects events
python examples/swe_smith/train_smith_agent.py \
  --agl-base-url http://localhost:8080 --agl-key "$AGL_KEY" \
  --model Qwen/Qwen3.5-9B

default_proxy.model_name must equal the trainer's actor_rollout_ref.model.path; a mismatch surfaces as "model not found" even when the vLLM endpoint is healthy.

The trainer-side configuration layers on verl's ppo_trainer Hydra config. The settings that decide correctness (reference template, from agentlightning/verl/config.yaml):

algorithm:
  enable_rollout_level_advantage: true      # Figure 4 baseline over rollouts, not rows
actor_rollout_ref:
  actor:
    policy_loss:
      loss_mode: per_rollout_mean           # Equation 16
  rollout:
    mode: async
agentlightning:
  agl_base_url: http://localhost:8080
  agl_key: ""
  rollout_timeout_seconds: 1800             # becomes activeDeadlineSeconds on the Job
  reward_fillna_value: 0.0                  # reward for a rollout that reported none
  max_ppo_update_times: null                # set to 2 for stability
  trace_aggregator:
    level: trajectory                       # transition | trajectory
    trajectory_max_prompt_length: 2048
    trajectory_max_response_length: 8192
  k8s:
    job_template_path: null
  async_rollout:
    enabled: false
    async_train_batch_size: null            # must be > data.train_batch_size

Note that the shipped YAML sets enable_rollout_level_advantage: true, but the code default when the key is absent is False (self.config.algorithm.get("enable_rollout_level_advantage", False) in trainer.py), and loss_mode falls back to "vanilla". A config assembled from scratch rather than composed from the package silently gets the sample-level behaviour the paper argues against.

How to develop with it¶

Wiring a new harness is three steps.

Give the Controller a way to start the agent. In local mode set agentlightning.local.agent_class to a fully qualified class and local.env_map to a mapping from environment variable name to a field of the rollout input (for example QUESTION: input.question). In k8s mode set agentlightning.k8s.job_template_path to a Jinja template that renders exactly one Kubernetes Job; the trainer reads the template text and ships it inside each rollout, and the Controller renders it per rollout. Use the yaml_escape filter on interpolated values. The Controller overwrites metadata.name, metadata.namespace, spec.backoffLimit: 0, spec.ttlSecondsAfterFinished, spec.activeDeadlineSeconds, and restartPolicy: Never, so do not rely on those from the template.

Point the agent at the proxy and report a reward. Reference template on the real interface, not executed here (the pattern from examples/gsm8k/gsm8k_agent.py and examples/calc_x/calc_agent.py, openai>=2.0.0,<3):

import os, httpx
from openai import AsyncOpenAI

client = AsyncOpenAI(base_url=os.environ["AGL_OPENAI_BASE_URL"],
                     api_key=os.environ["AGL_KEY"])
# ... run the harness unchanged; every call is captured as a model_request event ...
httpx.post(os.environ["AGL_EVENT_URL"],
           json={"event_type": "reward", "data": {"value": reward}},
           headers={"Authorization": f"Bearer {os.environ['AGL_KEY']}"},
           timeout=10.0).raise_for_status()

Pass datasets in memory. Agent Lightning does not use verl's data.train_files; run_ppo(config, train_dataset=..., val_dataset=...) takes any non-empty sequence of JSON-like objects, and each element becomes one rollout's input.

For local iteration, run the Controller with runner_type=local (local_runner.maximum_size: 50, poll_interval: 10) so rollouts are subprocesses on the same machine. The repository's own test suite is 2,160 lines and covers the parts most likely to break under a harness change: tests/verl/test_rollout_adapter.py (573 lines) for aggregation, tests/server/test_endpoints.py (401 lines) for the store contract, plus focused tests for rollout-level advantage and the per-rollout loss.

How to maintain it¶

Pin the commit, not main. There is no CHANGELOG, no migration guide, and no stated API-stability or deprecation policy anywhere in the repository or its documentation site as of commit e43cbf2. The only signals are the version string (1.0.1), the v1.0.0 and v1.0.1 tags, and a README note that v0.x lives on a separate branch. Treat every config key as breakable across patch releases.
Re-derive the trace-aggregation metrics after any harness, tokenizer, or chat-template change. Watch training/n_sample, training/n_rollouts, training/n_unmerged_rollouts, and training/n_trace_merge_mismatch_rows. A merge ratio that collapses means the chat template stopped round-tripping; the adapter logs the decoded previous and current traces to a W&B table for exactly this diagnosis.
Watch the drop counters. training/n_sample_dropped/marked (prompt over trajectory_max_prompt_length), training/n_truncated_sample (response truncated), training/n_sample_dropped/same_reward and /random (mini-batch alignment and the max_ppo_update_times cap), and training/n_skipped_empty_rows. A step can legitimately train on far fewer rows than it collected.
Track training/n_zero_adv_groups against training/n_groups. A rising ratio means the difficulty filter has drifted and the batch is mostly gradient-free.
Upgrade verl deliberately. The integration subclasses RayPPOTrainer, registers a custom policy loss through verl's register_policy_loss, and calls into verl.trainer.ppo.ray_trainer.compute_advantage; all three are internal surfaces. Nothing in the installed package enforces a verl version: verl is absent from [project] dependencies, and the verl>=0.7.1,<0.9.0 range sits in the verl-cpu optional group, which the surrounding pyproject.toml comment describes as a Pyright type-check environment and flatly states "This is not a training environment". The range is the maintainers' compatibility claim, not a constraint your training install will trip over; scripts/setup_verl.sh is what actually pins a training environment, and it accepts only 0.7.1 or 0.8.0. Pin verl explicitly on the training image and re-run the aggregation and per-rollout-loss tests after any bump.

How to run it in production¶

Topology. The paper's and repository's reference deployment is two machines. Machine A has cluster access, prepares images, and runs agl-controller; Machine B has the GPUs and runs both agl-server and the verl trainer. The Gateway address must be reachable from Machine A and from every rollout pod. Start order is server, controller, trainer, gated on GET /healthz returning 200. Use agl_server.agent_url when pods reach the Gateway by a different address than the Controller does.

Scale limits to size against. GET /api/rollouts returns at most limit rollouts (default 500) and the K8s reconciler queries with limit=500 per cycle, so more than 500 concurrently queuing-or-running rollouts will not all be visible in one pass. k8s_runner.max_jobs_per_minute defaults to 100, which caps cold-start ramp. k8s_runner.poll_interval is 5 seconds for the list-based reconcile; the watch loop propagates terminal states faster. The Gateway is one FastAPI process mutating module-level dictionaries on the event loop thread with no locks and no persistence, so it is both a single point of failure and a single point of throughput; the trainer deletes each rollout after reading its events, which bounds memory but leaves no history to inspect after the fact.⁶

Collocated asynchronous rollout. The paper's motivation is that fully asynchronous RL (separate rollout and update GPU pools, as in AReaL) needs more GPUs than a small team can afford, and it claims "roughly a 2x end-to-end speedup over synchronous RL while also using fewer GPUs" for the collocated variant; no measurement in the paper supports that figure.⁵ Set agentlightning.async_rollout.enabled: true and async_train_batch_size strictly greater than data.train_batch_size (the trainer raises otherwise); the documentation's starting point is 2x. The trainer keeps async_train_batch_size prompt groups active, consumes the first data.train_batch_size groups to complete, and carries the rest to the next step. A group is only consumable when all rollout.n siblings are terminal, which preserves GRPO group statistics. Before the update the trainer POSTs /proxy/pause, polls /proxy/state every 250 ms until inflight reaches zero, updates weights, then POSTs /proxy/resume; a request arriving while paused gets HTTP 429 with Retry-After and X-Agl-Paused: true, so the agent's HTTP client must retry. Watch training/async/proxy_drain_seconds and training/async/new_carry_over_age_max_steps.

Correct for the staleness this introduces. Carried-over rollouts were partly generated under an older policy version. The repository recommends verl's rollout correction with token-level importance sampling, algorithm.rollout_correction.rollout_is: token and rollout_is_threshold: 2. This is the same correction discussed in async and disaggregated RL systems and rollout reuse under policy lag. Keep default_proxy.include_log_probs: true or the rollout log probabilities the correction needs will not be recorded.

Reward hacking is an infrastructure problem here. In the coding-agent run the agents found the reference fix four ways: reading Git history for the gold commit, wget/curl from GitHub, pip downloading the package source, and urllib. The paper names two safeguards (Section 4.3.2): disabling Git commands and hiding the .git directory, and "a Kubernetes network policy that blocks general outbound network access and permits connections only to explicitly whitelisted services". No such policy is in the repository. What ships at e43cbf2 is command-level filtering inside the example harness, examples/swe_smith/agents/smith_agent.py, whose own docstring says "Network/install/tamper blocks are a code backstop; the authoritative fix is a default-deny egress NetworkPolicy".¹¹ Treat the cluster-side allowlist as something to write, not to inherit: the regexes match command position and named binaries, so an unlisted fetcher or a renamed one gets through, and then the reward signal is corrupt. This is the same containment problem as agent sandboxing and isolation, and the training-integrity view of it, including the reward-distribution statistics that expose a contaminated task after the run, is RL environment sandbox escape.

Authentication. key: "" disables auth entirely and only logs a warning at startup. The Gateway proxies to your inference servers and accepts arbitrary event writes, so set a real key and keep the Gateway off any untrusted network.

Reported results, with their setups. Search agent: Llama-3.2-3B-Instruct, GRPO, trained on the HotpotQA training split, evaluated on 50 examples from each of HotpotQA, 2WikiMultiHopQA, MuSiQue, Bamboogle, TriviaQA, and Natural Questions, batch size 512, 4 rollouts per prompt, exact match reward; validation reward 25.1% to 41.7% (Section 4.1). Instruction-following agent: Qwen3-4B-Instruct-2507, RLOO, the LLM-in-Sandbox harness and an 80/20 split of the Instruction Pre-Training dataset, batch size 8, 8 rollouts per prompt; validation reward 51.9% to 70.2% (Section 4.2). Coding agent: Qwen3.5-9B with mini-SWE-agent on roughly 6,000 filtered SWE-smith tasks, GRPO; SWE-bench Verified 41.8% to 56.4% at step 208 (Section 4.3.3). The paper never states the hardware; the repository's example page gives 4x B200, one node, and the training script sets train_batch_size: 16, rollout.n: 8, ppo_mini_batch_size: 16, lr: 1e-6, clip_ratio_low/high: 0.2/0.28, max_ppo_update_times: 2.³ The ablation behind the recommended configuration is narrower than it looks: rollout-level advantage alone scored 33.1% validation reward against a 35.0% sample-level baseline, and only the combination with rollout-level normalisation reached 38.2%.⁴

Failure modes¶

Merge ratio collapse after a tokenizer or template change. Prompts stop being exact token prefixes, every call becomes its own row, prefix computation is repeated, and the effective batch composition shifts. Detect with training/n_unmerged_rollouts and the merge-mismatch W&B table, not with loss curves.
Sample-level statistics smuggled in by a hand-built config. enable_rollout_level_advantage and loss_mode default to the sample-level behaviour when absent. Compose from agentlightning/verl/config.yaml and assert both values at startup.
A harness that streams. forward_request raises HTTP 400 on stream: true. Any harness that streams tokens for a UI must be given a non-streaming path before it can be trained.¹²
A backend that cannot return token IDs. The proxy sets return_token_ids: true; without it response_token_ids is empty, every triplet is dropped, and get_train_data_batch raises RuntimeError("get_train_data_batch emitted zero training rows.").
Crashed rollouts scored as failures. reward_fillna_value defaults to 0.0, so a rollout that died from an OOM, a pulled image, or a network partition is indistinguishable from one that genuinely failed the task, and a group where every sibling crashed becomes a silent zero-variance group. Alert on training/n_rollouts_w_reward divided by training/n_rollouts, not on the reward mean.
Gateway restart loses all state. The store is process-local memory. A restart mid-step orphans every in-flight rollout; the Controller then marks running rollouts failed because the Gateway no longer knows them.⁶
No retry for a failed rollout. Jobs are created with backoffLimit: 0 and attempt_id is hard-coded to "0" everywhere in both reconcilers; the attempt machinery in the schema is unused at this commit.⁷ A pod evicted by the scheduler is a lost sample, not a retried one.
Agents without a retrying HTTP client stall async training. A pause returns HTTP 429; a client that treats that as fatal fails the rollout at every weight update.
Long-tail rollouts hitting rollout_timeout_seconds. The Job's activeDeadlineSeconds kills the pod; the rollout goes FAILED with whatever partial events it recorded, and its group's reward gets the fillna value. The executed model above puts 3.5% of groups over a 5,400-second deadline on a lognormal tail with a 549-second mean.
Egress left open on rollout pods. The repository ships no NetworkPolicy manifest, so the deny-by-default egress rule the example harness defers to has to be written by the operator; without it the coding agent will fetch the upstream fix and the reward signal becomes noise that looks like fast learning.

References¶

Agent Lightning v1.0: Towards Harnessed Agentic RL (arXiv 2608.17528, 18 August 2026): https://arxiv.org/abs/2608.17528
Agent Lightning repository (MIT, read at commit e43cbf289e92385e0589e4113fdcfdcb822aebb9): https://github.com/microsoft/agent-lightning
Agent Lightning documentation site: https://microsoft.github.io/agent-lightning/stable/
Agent Lightning: Train ANY AI Agents with Reinforcement Learning (the v0.x paper, arXiv 2508.03680): https://arxiv.org/abs/2508.03680
vLLM blog, returning token IDs via the OpenAI-compatible API (the retokenisation-drift fix this depends on): https://blog.vllm.ai/2025/10/22/agent-lightning.html
verl Uni-Agent (proxy-based training, buffered token replacement, sandbox services): https://github.com/verl-project/uni-agent
Polar: Agentic RL on any harness at scale (arXiv 2605.24220): https://arxiv.org/abs/2605.24220
AReaL 2.0 / next-generation agentic RL systems (arXiv 2607.01120): https://arxiv.org/abs/2607.01120
mini-SWE-agent (the harness used for the coding-agent experiments): https://github.com/SWE-agent/mini-swe-agent
verl rollout correction (token-level importance sampling for carried-over rollouts): https://verl.readthedocs.io/en/latest/algo/rollout_corr.html

The paper gives the figure five times: "approximately 3,500 lines of code" in the abstract, Section 1, Section 3, and Related Work, and "an approximately 3,500-line framework" in the Conclusion. The repository README repeats it as "~3,500 lines of code". At commit e43cbf289e92385e0589e4113fdcfdcb822aebb9, find agentlightning -name '*.py' | xargs wc -l totals 4,758 lines across 27 files (largest: rollout_adapter.py 838, trainer.py 764, agl_rollout_manager.py 661). The v0.x branch (pyproject.toml version 0.3.1) totals 30,563 lines across 93 files under the same package. The 3,500 figure is plausibly the count at the v1.0.0 tag; it is not the count at current main, and the difference is flagged rather than resolved here. ↩
arXiv 2608.17528 Section 2.2 makes the advantage attribution: "verl Uni-Agent [12] and Polar [14] compute advantage at the rollout level, while slime [10] and AReaL [13] compute it at the sample level." Section 2.3 makes a separate and narrower one for loss normalisation: "slime [10] implements a rollout-level token-mean loss, which first pools all response tokens of a rollout together and then averages uniformly over rollouts", which is Equation 16, the form the paper then adopts ("We therefore prefer the rollout-level token-mean loss in Equation 16"). The paper says nothing about verl Uni-Agent's or Polar's loss normalisation, so the Section 2.2 line-up cannot be carried over to Section 2.3, and slime is on the same side as Agent Lightning on normalisation while on the opposite side on advantage. ↩↩
arXiv 2608.17528 Sections 4.3.1 and 4.3.3. Two things to keep straight. First, the SWE-bench Verified figures (41.8% to 56.4%) are from the step-208 checkpoint of the "Rollout-level Advantage + Rollout-level Norm" run, whereas the highest observed validation reward in Figure 9 is 38.2% at step 128; the paper does not report SWE-bench Verified for step 128 or for the other two variants, so the headline gain is not directly comparable to the ablation curve. Second, the abstract describes the gain as "an absolute 14.6% gain" while the Conclusion writes "a gain of 14.6 percentage points"; 56.4 minus 41.8 is 14.6 percentage points, so the Conclusion's phrasing is the correct one. Hardware is absent from the paper entirely; docs/75-example-coding-agent.md in the repository gives "4x B200", and examples/swe_smith/train_smith_agent.py sets trainer.n_gpus_per_node: 4, trainer.nnodes: 1. Data filtering: 59,136 SWE-smith tasks, minus 18,033 with an empty problem statement, minus 1,265 with a missing problem branch, minus tasks needing more than 200 tests, then a four-rollout Qwen3.5-9B difficulty probe keeping mixed-outcome tasks (about 5,000) plus 1,000 all-fail tasks, for "approximately 6,000 training examples and 400 test examples" (the paper's wording; it does not call the 400 a validation split). ↩
arXiv 2608.17528 Section 4.3.3 and Figure 9. Three variants on the same GRPO objective. The paper reports that the combined variant "produces the highest observed validation reward, reaching 38.2% at step 128, compared with 35.0% for the baseline and 33.1% when only the rollout-advantage fix is applied". Only the 38.2% is pinned to a step; the 35.0% Sample-level Advantage and 33.1% Rollout-level Advantage figures are given without one and should not be read as step-128 readings of those two curves. The principled advantage fix applied alone therefore scored 1.9 points below the sample-level baseline; the paper attributes this to a policy-entropy increase that the rollout-level loss normalisation then controls. Both settings are needed together, which is why shipping only enable_rollout_level_advantage: true is worse than shipping neither. ↩
arXiv 2608.17528 Section 3.1 states "In our experiments, collocated async RL achieves roughly a 2x end-to-end speedup over synchronous RL while also using fewer GPUs." No table, figure, or experimental setup in the paper supports this number: Figures 7 through 10 report reward, entropy, and merge statistics only, and no timing measurement appears anywhere in the paper or its appendix. The claim is recorded here as unsupported by the presented evidence, and the queueing model in the executed block is this page's own construction, not a reproduction of it. ↩
arXiv 2608.17528 Section 3 states the control plane "coordinates durable rollout state, external execution, partial failures, and resource usage". The implementation at e43cbf2 is agentlightning/server/store.py: four module-level Python objects (_rollouts, _events, _models, _terminal_order) whose own docstring reads "In-memory server state — single-threaded, no locks, plain dict/list." There is no persistence layer, no write-ahead log, and no replication; the v0.x line, by contrast, shipped SQLite and MongoDB store backends. AglRolloutManager.enqueue_and_wait_until_completed additionally issues DELETE /api/rollouts/{id} for each rollout after reading its events. "Durable" in the paper is best read as describing the declarative rollout object model rather than any storage guarantee. ↩↩
arXiv 2608.17528 Section 3 states "generation attempts are recorded and resolved explicitly". The schema does carry attempt_id on every event, last_attempt_id on the rollout status, and an attempt segment in both the proxy and event URLs. At e43cbf2 no code path ever produces a value other than DEFAULT_ATTEMPT_ID = "0": k8s_reconciler.build_job_spec hard-codes it into the Job labels and both injected URLs and sets spec["backoffLimit"] = 0, and local_reconciler does the same. There is exactly one attempt per rollout, and a failed attempt is not retried. ↩
arXiv 2608.17528 Table 1 lists nine endpoints. The code at e43cbf2 registers those nine plus six more: GET /api/rollouts/terminal (cursor pagination over an append-only completion log), DELETE /api/rollouts/{rollout_id}, POST /proxy/pause, POST /proxy/resume, GET /proxy/state, and an unauthenticated GET /healthz. The proxy route is also broader than the table: POST /proxy/rollout/{rollout_id}/attempt/{attempt_id}/mode/{mode}/openai/v1/{upstream_path:path} accepts chat/completions or completions. GET /api/rollouts/{rollout_id}/events additionally takes ?event_type= and ?format=triplet. ↩
arXiv 2608.17528 Section 3.2 places the deduplication of retried model calls in the trainer: "when the Customized Trainer assembles training samples, it deduplicates model_request events that share the same prompt". In the code it runs in the Gateway, in _dedupe_model_requests_by_prompt_token_ids inside agentlightning/server/routes/events.py, applied only when the caller passes ?format=triplet to GET /api/rollouts/{id}/events. The behaviour matches the paper (keep the last index per prompt_token_ids key); the component that performs it does not. The key is the prompt token ids, so two retries that retokenised differently are not deduplicated. ↩
arXiv 2608.17528 Equation 16 defines the rollout-level token-mean loss with an outer 1/R over the R rollouts in the batch. normalize_advantages_by_rollout in agentlightning/verl/per_rollout_loss.py divides each row's advantage by rollout_token_counts[rollout_id] * num_trained_rows, and trainer.py passes num_trained_rows=len(batch), the number of training rows, not the number of distinct rollouts. Relative weighting across rollouts is therefore exactly equal, which is the property the paper and the repository documentation both claim ("normalizing so every rollout carries equal weight regardless of its sample count"), but the global scale is Equation 16 multiplied by n_rollouts / n_rows, a factor that varies batch to batch with the merge ratio (2.41 on average in the paper's coding run). Whether verl's downstream gradient accumulation and the dp_size multiplication in compute_policy_loss_per_rollout_mean cancel this factor was NOT verified here: verl is not installed in this environment, so only the numpy model of the formula was executed. ↩
arXiv 2608.17528 Section 4.3.2 lists four observed reward-hacking routes (Git history, wget/curl, pip, Python networking libraries such as urllib) and "two safeguards": disabling Git commands plus hiding .git, and "a Kubernetes network policy that blocks general outbound network access and permits connections only to explicitly whitelisted services". The paper does not describe that policy any further, and in particular does not say the allowlist is the API Gateway alone. At e43cbf2 the repository contains 27 YAML files and zero with kind: NetworkPolicy, so nothing enforcing egress ships. The shipped control is regex action filtering in examples/swe_smith/agents/smith_agent.py: _GIT_INVOKE_RE and _GIT_ACCESS_RE (git in command position, --git-dir, --work-tree, .git), _NET_FETCH_RE (curl, wget, httpie, http, https, aria2c, scp, sftp, rsync, nc, ncat, netcat, telnet), _PKG_INSTALL_RE (pip, pip3, conda, mamba, easy_install, uv, and python -m pip), _PY_NET_RE (urllib.request, urlopen, requests.get/post/put/head/Session, httpx., socket.socket, socket.create_connection, urllib3), and _TEST_TAMPER_RE for writes to conftest.py, pytest.ini, tox.ini, setup.cfg, pyproject.toml, sitecustomize.py, usercustomize.py, and .pth. Blocked actions return a message to the model rather than terminating the rollout. The test-harness-tampering route is the code's own addition; the paper does not list it. ↩
The repository README claims agents train "with ZERO changes". Three constraints qualify that at e43cbf2. forward_request in agentlightning/server/proxy.py raises HTTPException(status_code=400, detail="Streaming responses are not supported") when the body carries stream: true. ProxyRouter.prepare_body replaces the harness's model and temperature with the Gateway's configured values and forces return_token_ids: true plus, in train mode and only when default_proxy.include_log_probs is true (default true), logprobs: true; the repository's own Gateway documentation confirms this, stating that verl's temperature settings "are not used for proxied requests because the proxy replaces them automatically". And docs/35-asynchronous-training.md requires that "Agents should use a retrying OpenAI or HTTP client" because a paused Gateway answers HTTP 429. A harness that streams, depends on a per-call temperature, or does not retry needs changes before it can be trained. ↩↩↩