Skip to content
Markdown

Compaction-aware RL

Scope: reinforcement learning for agents that compress their own interaction history during a rollout, so that the horizon a policy can be trained on stops being bounded by the model's context window. This page covers what a compaction boundary does to the shape of an RL sample, the three published ways of assigning advantage across segments and which of them is measurably wrong, the loss-normalisation and discount corrections that compaction forces, the rollout and infrastructure cost, and an audit of the published gains. It is the training-time counterpart to when to compact and to the tool-level design in agentic context management; the algorithms it modifies are catalogued in GRPO variants.

Three NumPy and stdlib blocks below are executed and asserted in this page (Python 3.11, NumPy 2.x). They reconstruct the segment-count bias from first principles, verify what CompactionRL's cross-trajectory correction does and does not fix against a reference GAE over the concatenated rollout, and recompute the published tables of arXiv 2509.13313 (ReSum), 2510.06727 (SUPO), 2607.05378 (CompactionRL) and the AutoCompact project page. The benchmark numbers are the papers'; the standard errors, the significance ratios and the audit conclusions are ours and are derived in the blocks. No training run was reproduced.

What it is

A long-horizon agent that summarises its history mid-task produces a rollout that is no longer one sequence. It is a chain of segments, each beginning from the task prompt plus the previous summary, each ending either at a new summary or at the final answer. The environment still returns exactly one reward, at the very end.

That breaks an assumption every standard LLM RL recipe rests on: that one rollout is one training sample with one reward. Compaction-aware RL is the family of modifications that make policy gradient work on the chain instead.

Method Trigger for compaction Who writes the summary Optimisation unit Advantage source
ReSum-GRPO (arXiv 2509.13313) context limit approached a separate frozen tool model (ReSumTool-30B) segment group-relative, broadcast from the rollout
SUPO (arXiv 2510.06727) context length crosses L the trained policy itself segment group-relative over the G rollouts
CompactionRL (arXiv 2607.05378) remaining budget below T_comp the trained policy itself token a critic, with a cross-segment discount correction
AutoCompact (project page, 2026-07-30) the policy calls compact() the trained policy itself token GRPO, trajectory advantage on every token

The four differ on two axes that matter operationally. The first is whether the summariser is inside the policy being trained: ReSum keeps it outside, the other three pull it in, so the gradient reaches the summary tokens. The second is whether the compaction timing is learned: only AutoCompact and (through the tool-call decision) ACM train it, while ReSum, SUPO and CompactionRL all keep a token-count threshold and train only the behaviour around it.

SUPO gives the cleanest formal statement. It extends the multi-turn MDP with a summarisation-augmented transition: when the context exceeds a threshold L, the next state becomes the initial prompt plus the model's own summary rather than the appended history. Its Theorem 3.2 then shows the policy gradient of that MDP decomposes into a sum over the I+1 segments, each of which has the shape of an ordinary single-trajectory rollout. That is the load-bearing result for infrastructure: it means an existing verl-style or slime-style trainer needs no new gradient path, only a different way of cutting rollouts into samples and a different way of assigning advantage to them.

Why use it

  • The training horizon stops being the context window. SUPO defines an effective length L_RL x (S+1) for S summarisations. Its CodeGym configuration trains at a 4K working context with up to 8 segments, reaching a 32K effective horizon on a model whose training context is 4K.
  • Inference-time compaction is a distribution the base model has never seen. Reasoning from a summary of your own history is a different task from reasoning from raw history. CompactionRL's Table 2 shows base GLM-4.5-Air scoring 59.8 on SWE-bench Verified under compacted evaluation, and standard PPO with no compaction in training raising that only to 62.5 while compaction-aware training reaches 66.8. Training on the compacted distribution is where the gain is.
  • Summary quality is a first-class lever, and it is measurable independently. CompactionRL fixes the execution agent and swaps only the summariser: 49.0 with Qwen3-30B-A3B, 50.5 self-summarising, 55.5 with Qwen3.5-27B. That is a 6.5-point spread from a component many teams treat as a preprocessing step.
  • Rollout cost falls even when accuracy does not move. Every token generated after a compaction attends to a summary rather than to the full history, which shortens prefill and shrinks the KV working set for the rest of the episode. This matters more than the accuracy delta on a fleet that is rollout-bound, which long-horizon RL usually is.

When to use it (and when not)

Use it when episodes routinely exhaust the working context before the task finishes, when you already run RL on the agent, and when you can afford a second evaluation configuration (compacted and non-compacted) because the two diverge.

Do not use it when:

  • Your episodes fit. If compaction never fires, you have added a distribution shift and a bug surface for nothing.
  • You have not first tried a better summariser. On CompactionRL's own numbers, swapping in a stronger off-the-shelf summariser buys +5.0 points on SWE-bench Verified against the base model's self-summarising score of 50.5, and the entire RL campaign buys +5.5. Block 3 asserts both. Run the swap experiment first; it costs one evaluation sweep instead of a training campaign.
  • You cannot double your evaluation budget. CompactionRL's trained models get worse at single-window inference: GLM-4.7-Flash drops from 47.5 to 43.7 on SWE-bench Verified with compaction disabled. If some of your traffic never compacts, you now have two regimes to track.
  • Your reward is dense or per-step. The whole difficulty here comes from one terminal reward having to reach across segment boundaries. With per-step reward most of it evaporates.
  • You want the timing learned and are not prepared to build an annotation pipeline. AutoCompact needed a judge (GPT-5.5-Codex) reviewing base-model actions step by step to produce 1,052 cold-start examples, because base models essentially never invoke a compaction tool on their own even when told how.

Architecture

flowchart TB
  subgraph ROLL["Rollout collection"]
    P["task prompt s1"] --> E1["execution segment 1<br/>think / tool / observe"]
    E1 -->|"context crosses threshold"| S1["summary segment 1<br/>policy writes the summary"]
    S1 --> E2["execution segment 2<br/>prompt + summary + recent turns"]
    E2 -->|"threshold again"| S2["summary segment 2"]
    S2 --> E3["execution segment K"]
    E3 --> V["verifier: one terminal reward R"]
  end
  V -->|"R shared by every segment"| ADV{"advantage across segments"}
  ADV -->|"normalise over G rollouts,<br/>broadcast (ReSum, SUPO)"| B1["group-relative"]
  ADV -->|"normalise over all segments<br/>(SUPO ablation)"| B2["segment-count bias"]
  ADV -->|"critic + (gamma*lambda)^N_after<br/>(CompactionRL)"| B3["cross-trajectory GAE"]
  B1 --> LOSS["token-level normalised<br/>PPO / GRPO loss"]
  B2 -.->|"measurably worse"| LOSS
  B3 --> LOSS

How to use it

Block 1: what a compaction boundary does to the sample count

Two things go wrong the moment one rollout yields a variable number of samples. The baseline that group-relative methods subtract gets pulled by rollouts that happened to compact more, and the loss normaliser hands those rollouts more gradient mass. Both are arithmetic, and both are visible without a GPU.

import numpy as np

# ============ 1. what a compaction boundary does to the sample count ==========
# One prompt, G rollouts. Compaction splits rollout g into K_g segments that all
# carry the same terminal reward R_g. Two rollouts fail, two succeed, but the
# rollouts that compacted more produce more segments.
K = np.array([1, 4, 1, 4])          # segments per rollout
R = np.array([0.0, 0.0, 1.0, 1.0])  # terminal reward per rollout
G = len(K)

# (a) rollout-group advantage: normalise over the G rollout rewards, then
#     broadcast one number to every token of every segment of that rollout.
#     This is SUPO eq. (3) and ReSum-GRPO's advantage broadcasting.
A_rollout = (R - R.mean()) / R.std()

# (b) trajectory-group advantage: expand R by segment count first, then
#     normalise over the expanded list. This is SUPO's ablated eq. (4).
R_expanded = np.repeat(R, K)
A_traj = (R - R_expanded.mean()) / R_expanded.std()

assert R_expanded.size == K.sum() == 10
assert np.isclose(R.mean(), 0.5) and np.isclose(R_expanded.mean(), 0.5)
# With a symmetric split the two baselines coincide, so make the split asymmetric:
K2 = np.array([1, 1, 4, 4])          # now the SUCCESSES are the ones that compacted
R2_expanded = np.repeat(R, K2)
assert np.isclose(R2_expanded.mean(), 0.8)   # baseline pulled up by the winners
A_traj2 = (R - R2_expanded.mean()) / R2_expanded.std()

# The failures are punished harder and the successes rewarded less, purely
# because the successes happened to compact more often.
assert A_traj2[0] < A_rollout[0]
assert A_traj2[2] < A_rollout[2]
print("rollout-group advantage   :", np.round(A_rollout, 4))
print("trajectory-group (sym K)  :", np.round(A_traj, 4))
print("trajectory-group (asym K) :", np.round(A_traj2, 4))
print("baseline shift            :", round(float(R2_expanded.mean() - R.mean()), 4))

# ============ 2. segment-count bias in the loss normaliser ====================
# Same four rollouts. Token counts per segment differ; the summary segments are
# short, the execution segments long.
seg_tokens = {
    0: [900],
    1: [700, 200, 650, 150],
    2: [850],
    3: [600, 180, 720, 160],
}
adv = {g: A_rollout[g] for g in range(G)}

def weight_per_rollout(normaliser):
    """Total |dL/dA| mass each rollout contributes under a given normaliser."""
    if normaliser == "segment":
        # mean over segments of (mean over that segment's tokens)
        n_seg = sum(len(v) for v in seg_tokens.values())
        return {g: len(seg_tokens[g]) / n_seg for g in range(G)}
    if normaliser == "token":
        # single mean over every optimised token in the batch
        n_tok = sum(sum(v) for v in seg_tokens.values())
        return {g: sum(seg_tokens[g]) / n_tok for g in range(G)}
    raise ValueError(normaliser)

w_seg = weight_per_rollout("segment")
w_tok = weight_per_rollout("token")

# Under segment-level averaging the two compacting rollouts take 80% of the
# gradient mass while holding 65.8% of the tokens.
assert np.isclose(w_seg[1] + w_seg[3], 0.8)
assert round(w_tok[1] + w_tok[3], 3) == 0.658
# Token-level normalisation removes the segment-count amplification: each
# compacting rollout drops from 4x to about 1.7x the weight of a single-segment one.
assert np.isclose(w_seg[1] / w_seg[0], 4.0)
assert round(w_tok[1] / w_tok[0], 3) == 1.889
print("segment-normalised weights:", {g: round(w_seg[g], 4) for g in range(G)})
print("token-normalised weights  :", {g: round(w_tok[g], 4) for g in range(G)})

Output:

rollout-group advantage   : [-1. -1.  1.  1.]
trajectory-group (sym K)  : [-1. -1.  1.  1.]
trajectory-group (asym K) : [-2.  -2.   0.5  0.5]
baseline shift            : 0.3
segment-normalised weights: {0: 0.1, 1: 0.4, 2: 0.1, 3: 0.4}
token-normalised weights  : {0: 0.1761, 1: 0.3327, 2: 0.1663, 3: 0.3249}

Two things to take from this. First, the two normalisations coincide exactly when segment counts are uncorrelated with reward, which is why the bug is easy to miss on a smoke test and appears in production where longer, harder, more-compacted rollouts fail more often. When the correlation is there, the baseline moves by 0.3 and the advantages go from a clean plus-or-minus 1 to a lopsided minus 2 against plus 0.5. Second, segment-level averaging gives the compacting rollouts 80% of the gradient mass on 65.8% of the tokens, and token-level normalisation is what removes that.

Both papers that measured this reached the same conclusion from opposite directions. SUPO ablates its rollout-group normalisation against the trajectory-group version and loses 4.0 points on BrowseComp-Plus (53.0 to 49.0); on CodeGym the trajectory-group variant scores 42.1, which is below the plain GRPO baseline at 44.5, so getting the normaliser wrong turns the whole method into a regression. CompactionRL ablates its token-level loss and drops from 66.8 to 60.0 on SWE-bench Verified, the largest single ablation in that paper, against 63.0 for removing the GAE correction.

CompactionRL avoids the baseline problem entirely by using PPO with a critic and a group size of 1, on the argument that fixed-size reward groups do not survive variable segment counts. That is a defensible reading, but note the cost: a critic initialised from the policy checkpoint, 50 steps of value pretraining before RL, and two value updates per policy update.

Block 2: the discount across a compaction boundary

If each segment is optimised as its own trajectory, the terminal reward lands at the end of every segment, so an action in segment 1 looks as close to the outcome as an action in the last segment. CompactionRL's eq. (14) corrects this by scaling segment s by (gamma*lambda)^N_after, where N_after counts optimised tokens generated after that segment. The claim is that this restores the reward's true distance to the outcome. It does, exactly; it does not restore the rest of the advantage.

import numpy as np

rng = np.random.default_rng(7)
GAMMA, LAM = 1.0, 0.97
GL = GAMMA * LAM
BOUNDS = [0, 6, 11, 18]          # 3 segments over 18 optimised tokens
T = BOUNDS[-1]
V = rng.normal(0.4, 0.15, size=T)   # a critic's token values, arbitrary but fixed


def gae_concatenated(reward):
    """GAE over the rollout as one uninterrupted trajectory."""
    delta = np.empty(T)
    for t in range(T):
        v_next = V[t + 1] if t + 1 < T else 0.0
        r = reward if t == T - 1 else 0.0
        delta[t] = r + GAMMA * v_next - V[t]
    adv = np.empty(T)
    running = 0.0
    for t in range(T - 1, -1, -1):
        running = delta[t] + GL * running
        adv[t] = running
    return adv


def gae_segment_local(reward):
    """Each segment optimised on its own, terminal reward pinned to its last token."""
    adv = np.empty(T)
    for a, b in zip(BOUNDS[:-1], BOUNDS[1:]):
        running = 0.0
        for t in range(b - 1, a - 1, -1):
            v_next = V[t + 1] if t + 1 < b else 0.0
            r = reward if t == b - 1 else 0.0
            running = (r + GAMMA * v_next - V[t]) + GL * running
            adv[t] = running
    return adv


def trajectory_position_correction(adv):
    """CompactionRL eq. (14): scale segment s by (gamma*lambda)^(tokens after s)."""
    out = adv.copy()
    for a, b in zip(BOUNDS[:-1], BOUNDS[1:]):
        out[a:b] *= GL ** (T - b)
    return out


# GAE is affine in the terminal reward, so differencing isolates its coefficient.
coef_true = gae_concatenated(1.0) - gae_concatenated(0.0)
coef_local = gae_segment_local(1.0) - gae_segment_local(0.0)
coef_fixed = trajectory_position_correction(gae_segment_local(1.0)) \
    - trajectory_position_correction(gae_segment_local(0.0))

# The true coefficient is the discount over the distance to the real final token.
assert np.allclose(coef_true, GL ** (T - 1 - np.arange(T)))
# Uncorrected, every segment restarts the clock: token 0 is credited as if the
# task ended 5 tokens later, not 17. That is a 1.44x over-credit.
assert np.isclose(coef_local[0], GL ** (BOUNDS[1] - 1))
assert round(float(coef_local[0] / coef_true[0]), 3) == 1.441
# The correction makes the reward coefficient exact at every token.
assert np.allclose(coef_fixed, coef_true)

# It does not make the whole advantage exact: the local pass bootstraps V to zero
# at every segment boundary, and eq. (14) rescales that error rather than undoing it.
resid_true = gae_concatenated(0.0)
resid_fixed = trajectory_position_correction(gae_segment_local(0.0))
last = BOUNDS[-2]
assert np.allclose(resid_fixed[last:], resid_true[last:])   # final segment is exact
assert not np.allclose(resid_fixed[:last], resid_true[:last])
gap = float(np.abs(resid_fixed[:last] - resid_true[:last]).max())
rel = float((np.abs(resid_fixed[:last] - resid_true[:last])
             / np.abs(resid_true[:last])).max())
assert 0.04 < gap < 0.05 and 0.3 < rel < 0.4
print("reward-coefficient error after correction:",
      float(np.abs(coef_fixed - coef_true).max()))
print("value-term gap on pre-final segments     :",
      round(gap, 4), f"({rel:.1%} relative)")
print("over-credit of the first token, uncorrected:",
      round(float(coef_local[0] / coef_true[0]), 3))

# ---- how hard the correction actually bites at production lengths -----------
# CompactionRL uses length-adaptive GAE, lam = 1 - 1/(alpha*l) with alpha = 1.5.
# l ("response length") is not pinned down in the paper, so sweep it and report
# the surviving weight of the first segment of a 4-segment 64k-budget rollout.
ALPHA = 1.5
tokens_after_first_segment = 45_000        # our assumption, not the paper's
for l in (10_240, 32_000, 64_000):
    lam = 1.0 - 1.0 / (ALPHA * l)
    print(f"  l={l:>6}: lambda={lam:.7f} -> first-segment weight "
          f"{lam ** tokens_after_first_segment:.4f}")
assert (1 - 1 / (ALPHA * 10_240)) ** tokens_after_first_segment < 0.06
assert (1 - 1 / (ALPHA * 64_000)) ** tokens_after_first_segment > 0.62

Output:

reward-coefficient error after correction: 3.3306690738754696e-16
value-term gap on pre-final segments     : 0.0447 (33.5% relative)
over-credit of the first token, uncorrected: 1.441
  l= 10240: lambda=0.9999349 -> first-segment weight 0.0534
  l= 32000: lambda=0.9999792 -> first-segment weight 0.3916
  l= 64000: lambda=0.9999896 -> first-segment weight 0.6258

Three consequences worth carrying into a design review.

The correction is exact for the reward and approximate for the value. The reward coefficient matches the concatenated reference to floating-point precision at every token. The value terms do not, because the local pass bootstraps the critic to zero at each segment boundary and eq. (14) rescales that error rather than removing it. On this small example the residual is 33.5% relative on the pre-final segments and exactly zero on the last one. If your critic is well fit, that residual is the dominant remaining bias.

The uncorrected version over-credits early segments, which is the failure the paper is guarding against. At these lengths it is 1.44x; at production segment lengths the ratio is (gamma*lambda)^-N_after, which grows without bound.

The size of the correction depends entirely on how you read l. CompactionRL specifies length-adaptive GAE with lambda = 1 - 1/(alpha*l) and alpha = 1.5 but does not pin down whether l is the per-response cap (10,240 tokens) or the trajectory length. The sweep above shows the first segment of a four-segment rollout retaining 5.3% of its weight under the first reading and 62.6% under the third. That is not a detail: under the aggressive reading the earliest summary in an episode is learned roughly twenty times more slowly than the last one. If you implement this, log the effective lambda and the realised N_after distribution rather than trusting the default.

Block 3: reading the four published results honestly

import numpy as np


def se_diff(p1, n1, p2, n2):
    """Standard error of a difference of two independent binomial rates, in points."""
    return 100 * np.sqrt(p1 * (1 - p1) / n1 + p2 * (1 - p2) / n2)


# ============ SUPO, arXiv 2510.06727 Table 1 =================================
# CodeGym holds the effective budget equal; BrowseComp-Plus does not.
codegym = {"grpo": (44.5, 32, 32), "supo": (47.7, 4, 32)}       # acc, working K, effective K
bcp = {"grpo": (39.0, 64, 64), "supo": (53.0, 64, 192)}
assert codegym["grpo"][2] == codegym["supo"][2] == 32           # like-for-like
assert bcp["supo"][2] / bcp["grpo"][2] == 3.0                   # not like-for-like
assert codegym["supo"][1] * 8 == codegym["grpo"][1]             # 8x smaller window

# Eval-set sizes stated in the paper: 128 CodeGym, 100 BrowseComp-Plus.
cg = se_diff(0.445, 128, 0.477, 128)
bp = se_diff(0.390, 100, 0.530, 100)
assert round(cg, 1) == 6.2 and round(bp, 1) == 7.0
assert (47.7 - 44.5) / cg < 0.6      # +3.2 points is about half a standard error
assert (53.0 - 39.0) / bp > 1.9      # +14.0 points is about two
# The advantage-normalisation ablation is worse than plain GRPO on CodeGym.
assert 42.1 < codegym["grpo"][0]
print(f"CodeGym  +{47.7 - 44.5:.1f} pts, SE {cg:.1f} -> {(47.7 - 44.5) / cg:.2f} sigma, "
      f"effective length matched")
print(f"BC-Plus  +{53.0 - 39.0:.1f} pts, SE {bp:.1f} -> {(53.0 - 39.0) / bp:.2f} sigma, "
      f"effective length 3x the baseline")

# ============ ReSum, arXiv 2509.13313 Tables 1 and 2 =========================
# Rows: WebSailor-3B / 7B / 30B; columns GAIA, BrowseComp-zh, BrowseComp (Pass@1).
react = np.array([[25.6, 8.2, 3.3], [31.7, 13.2, 5.7], [45.0, 23.9, 12.8]])
resum_free = np.array([[35.3, 13.7, 6.8], [40.5, 17.2, 9.0], [47.3, 24.1, 16.0]])
grpo_react = np.array([[28.5, 11.8, 4.2], [34.0, 18.7, 5.8], [48.2, 23.3, 14.3]])
resum_grpo = np.array([[37.9, 20.5, 9.2], [42.4, 27.1, 12.3], [48.5, 33.3, 18.3]])

# The training-free headline reproduces to the decimal.
assert round(float((resum_free - react).mean()), 1) == 4.5
# The "further 8.2%" does not, under any of the three natural baselines.
readings = {
    "vs training-free ReSum": float((resum_grpo - resum_free).mean()),
    "vs GRPO on ReAct rollouts": float((resum_grpo - grpo_react).mean()),
    "vs untrained ReAct": float((resum_grpo - react).mean()),
}
assert [round(v, 2) for v in readings.values()] == [4.40, 6.74, 8.90]
assert all(abs(v - 8.2) > 0.6 for v in readings.values())
print("ReSum training-free mean gain:", round(float((resum_free - react).mean()), 2))
for k, v in readings.items():
    print(f"  ReSum-GRPO {k:<26}: {v:+.2f} pts")

# Two cells where ReSum-GRPO does not lead its own comparison set.
assert resum_grpo[0, 0] < 38.5          # 3B GAIA: GRPO-on-ReSum-rollouts scores 38.5
assert resum_grpo[2, 2] < 19.5          # 30B BrowseComp: MEM1-GRPO scores 19.5

# ============ CompactionRL, arXiv 2607.05378 Tables 1, 2, 3 ==================
# Table 1: same execution agent (GLM-4.7-Flash), summariser swapped.
swap = {"Qwen3-30B-A3B": 49.0, "GLM-4.7-Flash (self)": 50.5, "Qwen3.5-27B": 55.5}
# Table 2: the base model's compacted score is exactly the self-summarising row.
assert swap["GLM-4.7-Flash (self)"] == 50.5
trained = 56.0                                    # + CompactionRL, compacted (x4)
buy_by_swapping = swap["Qwen3.5-27B"] - swap["GLM-4.7-Flash (self)"]
buy_by_training = trained - swap["GLM-4.7-Flash (self)"]
assert (buy_by_swapping, buy_by_training) == (5.0, 5.5)

# Headline gains are against the base model under compacted evaluation.
air = {"base": (59.8, 21.4), "rl_1x": (62.5, 23.6), "rl_2x": (64.5, 23.6),
       "compaction": (66.8, 24.5)}
assert (round(air["compaction"][0] - air["base"][0], 1),
        round(air["compaction"][1] - air["base"][1], 1)) == (7.0, 3.1)
# Against RL trained at twice the context budget the same gains shrink sharply.
assert (round(air["compaction"][0] - air["rl_2x"][0], 1),
        round(air["compaction"][1] - air["rl_2x"][1], 1)) == (2.3, 0.9)

# Benchmark sizes: a random 200-instance SWE-bench Verified subset, all 89
# Terminal-Bench 2.0 tasks, mean of 2 runs.
swe_se = se_diff(0.598, 400, 0.668, 400)          # 2 runs x 200 instances
tb_se = se_diff(0.214, 178, 0.245, 178)           # 2 runs x 89 tasks
assert round(swe_se, 1) == 3.4 and round(tb_se, 1) == 4.5
assert 7.0 / swe_se > 2.0 and 3.1 / tb_se < 0.8
print(f"CompactionRL: swapping the summariser buys {buy_by_swapping:+.1f} pts, "
      f"the RL campaign {buy_by_training:+.1f}")
print(f"  SWE-bench Verified +7.0 pts, SE {swe_se:.1f} -> {7.0 / swe_se:.2f} sigma")
print(f"  Terminal-Bench 2.0 +3.1 pts, SE {tb_se:.1f} -> {3.1 / tb_se:.2f} sigma")
print(f"  vs RL at 2x context: +{air['compaction'][0] - air['rl_2x'][0]:.1f} and "
      f"+{air['compaction'][1] - air['rl_2x'][1]:.1f}")

# ============ AutoCompact (project page, no paper, no tables) ================
# The only numbers the page states as text rather than an unlabelled chart.
sft_trigger, rl_trigger = 0.443, 0.585
assert round(rl_trigger - sft_trigger, 3) == 0.142
print(f"AutoCompact trigger rate {sft_trigger:.1%} (SFT) -> {rl_trigger:.1%} (RL), "
      f"+{rl_trigger - sft_trigger:.1%} of tasks")

Output:

CodeGym  +3.2 pts, SE 6.2 -> 0.51 sigma, effective length matched
BC-Plus  +14.0 pts, SE 7.0 -> 2.01 sigma, effective length 3x the baseline
ReSum training-free mean gain: 4.5
  ReSum-GRPO vs training-free ReSum    : +4.40 pts
  ReSum-GRPO vs GRPO on ReAct rollouts : +6.74 pts
  ReSum-GRPO vs untrained ReAct        : +8.90 pts
CompactionRL: swapping the summariser buys +5.0 pts, the RL campaign +5.5
  SWE-bench Verified +7.0 pts, SE 3.4 -> 2.06 sigma
  Terminal-Bench 2.0 +3.1 pts, SE 4.5 -> 0.70 sigma
  vs RL at 2x context: +2.3 and +0.9
AutoCompact trigger rate 44.3% (SFT) -> 58.5% (RL), +14.2% of tasks

SUPO's two headline numbers answer two different questions. The CodeGym result is the interesting one and the paper undersells it: SUPO matches or beats GRPO at the same 32K effective budget while running an 8x smaller working window, which is a real memory and rollout-latency win. But at +3.2 points on a 128-task evaluation set it is about half a standard error, so treat it as "no worse" rather than "better". The BrowseComp-Plus +14.0 is statistically solid at roughly two standard errors on 100 tasks, but it compares a 192K effective budget against a 64K one. That number is a mixture of "summarisation helps" and "3x the effective context helps", and the paper does not separate them.

ReSum's training-free claim reproduces exactly and its trained claim does not. The 4.5-point average over ReAct comes out to 4.5 across the nine backbone-benchmark cells of Table 1. The abstract's "further 8.2% gain" does not match any of the three natural baselines in Table 2: 4.40 against training-free ReSum, 6.74 against GRPO on ReAct rollouts, 8.90 against untrained ReAct. Version v3 of the paper carries updated tables, and the abstract most likely was not recomputed. Cite the mechanism, not the 8.2.

Two cells also cut against the framing. On GAIA at 3B, ReSum-GRPO (37.9) trails plain GRPO run on ReSum rollouts (38.5), so the segmented objective is not what wins there. On BrowseComp at 30B, MEM1-GRPO scores 19.5 Pass@1 and 29.7 Pass@3 against ReSum-GRPO's 18.3 and 26.5, so the baseline the paper describes as poorly compatible wins that cell outright once it is also trained.

CompactionRL's honest comparison is not the one in the abstract. The +7.0 and +3.1 are against the base model under compacted evaluation. Against the more demanding baseline the paper itself runs, standard RL trained at twice the context budget, the gains are +2.3 and +0.9. On a 200-instance SWE-bench subset averaged over 2 runs the +7.0 is about two standard errors; the +3.1 on Terminal-Bench 2.0's 89 tasks is 0.70 of one, so it is not distinguishable from noise. The paper's real contribution is the ablation table, not the headline: token-level normalisation is worth 6.8 points and the GAE correction 3.8, which is direct evidence that the two corrections in Blocks 1 and 2 are the load-bearing parts.

AutoCompact is a project page, not a paper. There is no arXiv entry, no code, no weights and no numeric table; the accuracy figures live in charts whose axes are labelled only with a range. The two numbers it states in text are the trigger rates, 44.3% of tasks after SFT rising to 58.5% after RL. Its "+10.6% on average on SWE-bench Verified" is not broken down anywhere and cannot be checked. Use it for the mechanism (judge-guided step-level annotation to bootstrap a behaviour the base model will not emit, then GRPO applying the trajectory advantage to the compact() decision, the summary tokens and the continuation alike) and not for the numbers.

How to develop with it

The four papers converge on a rollout loop that is worth copying wholesale.

Cut at an atomic step, never inside one. CompactionRL treats each assistant-response and observation pair as atomic, so a tool call and its result are never separated by a compaction. SUPO goes further and discards the action-observation pair that pushed the context over the threshold, which bounds the working context to L + |v_sum| + L_A instead of the looser L + 2L_A + L_O + |v_sum| from its Proposition 3.1. That matters when observations are large: without the discard, a single oversized tool result can push the summary itself past the training context and get it truncated.

Keep a short verbatim tail. CompactionRL rebuilds the context as system prompt, summary template, and the last k = 2 steps verbatim, reducing k when needed to fit. The tail is what preserves exact current state (the file just edited, the command just run) that a summary paraphrases badly.

Mask rollouts that never finished. SUPO masks the gradient from rollouts that hit the step cap H or the summarisation cap S without producing a final answer. Without it, the objective suppresses long rollouts that were compacting well but ran out of budget, and the paper reports this collapsing the summarisation pattern. The ablation costs 9.0 points on BrowseComp-Plus (53.0 to 44.0).

Do not add a summary-quality reward. CompactionRL declines to, on the grounds that hand-designed summary metrics do not track which details actually mattered. The terminal task reward reaching the summary tokens is the whole point of putting the summariser inside the policy.

Pad the segment count to your mini-batch size. One rollout stage produces sum over b,j of (1 + I) trajectories, a number that varies per step. SUPO pads to a multiple of the mini-batch size with dummy zero-mask trajectories that contribute nothing to the update. This is the least interesting part of the design and the one most likely to be skipped and then debugged for a week.

Two reference configurations, both from the papers rather than from any run of ours:

SUPO (arXiv 2510.06727)
  CodeGym          Qwen2.5-32B-Instruct, batch 128, G=8, working 4K, S=7 (8 segments)
  BrowseComp-Plus  Seed-OSS-36B-Instruct, batch 32, G=8, working 64K, S=2 (3 segments)
  common           L = 0.95 * working context, H = 100, lr 1e-6 constant,
                   eps_low 0.20 / eps_high 0.28, no KL and no entropy loss

CompactionRL (arXiv 2607.05378)
  models           GLM-4.7-Flash (30B-A3B) at 64K, GLM-4.5-Air-SFT (106B-A30B) at 80K
  compaction       fires when remaining budget < 10,240 tokens, at most 3 per rollout,
                   keeps k=2 recent steps, response cap 10,240 tokens, 250 turns
  optimisation     PPO, group size 1, critic from the policy checkpoint,
                   50 steps value pretraining, 2 value updates per policy update,
                   policy lr 2e-6, critic lr 3e-6, length-adaptive GAE (alpha 1.5)
  framework        slime, Harbor evaluation environment, Terminus-KIRA scaffold

CompactionRL reports being deployed in the RL pipeline for GLM-5.2 (750B-A40B), which is the strongest available evidence that this class of method survives contact with a production training run.

How to maintain it

  • Version the summarisation instruction with the checkpoint. The policy is trained to respond to one specific v_sum prompt. Changing that string after training is a silent distribution shift, and it will show up as a quiet accuracy regression rather than an error.
  • Store the segment boundaries with the rollout. Advantage assignment, loss normalisation and the GAE correction all depend on them. Without them in the trace you cannot reproduce an update or diagnose one.
  • Re-derive the effective-length claim whenever the threshold moves. L_RL x (S+1) is only meaningful if L is still 95% of the working context and the discard rule still holds.
  • Keep both evaluation configurations in CI. Single-window and compacted scores move in opposite directions under this training; tracking only one hides half of what changed.
  • Track the segment-count distribution per training step. A drift toward more segments per rollout is the leading indicator of both the loss-normalisation bias and a summariser that has learned to stall.

How to run it in production

  • Budget the rollout stage, not the update stage. Long-horizon RL is rollout-bound, and compaction changes rollout cost in both directions: fewer tokens attended per step after a compaction, more LLM calls because each summary is a generation. Measure wall-clock per training step before and after, not just peak memory.
  • Meter compaction rate as a training-health signal. SUPO reports its CodeGym summarisation rate rising through training while conditional success on summarised rollouts also rises: that pair moving together is what you want. The rate rising while conditional success falls means the policy has learned to compact instead of to finish.
  • Serve with the same threshold you trained with. CompactionRL's models are worse without compaction than with it, because disabling it is a train-serve mismatch that also raises the overlong rate. Pin the threshold in the serving config next to the checkpoint.
  • Expect the summariser to be your quality floor. Before any of this, run the summariser-swap experiment: hold the execution agent fixed, vary only the summary model, and measure. A 6.5-point spread from that alone (CompactionRL Table 1) sets the bar the training campaign has to clear.
  • Watch for the trained model exploiting the compaction budget. With at most three compactions per rollout and a reward only at the end, a policy that cannot finish has no incentive to stop compacting. The overlong mask handles this during training; in serving you need a hard cap and an alert.

Failure modes

  • Trajectory-group advantage normalisation. The default when you naively hand segments to a GRPO trainer, and the one ablation in SUPO that turns a gain into a regression on CodeGym.
  • Sequence-level loss averaging over segments. Gives more-compacted rollouts gradient mass proportional to their segment count rather than their token count. Worth 6.8 points in CompactionRL's ablation.
  • Segment-local GAE with no boundary correction. Over-credits early actions by (gamma*lambda)^-N_after, which grows with the rollout.
  • Trusting the corrected advantage as exact. Block 2 shows the correction is exact for the reward term and leaves a value-bootstrapping residual on every pre-final segment.
  • Compacting inside an atomic step. Splitting a tool call from its observation puts the model in a state it will never see at inference.
  • No overlong mask. Suppresses exactly the long, well-summarised rollouts you are trying to learn from.
  • Changing the summarisation prompt after training. Silent, and it degrades precisely the behaviour you paid to train.
  • Reporting only the compacted evaluation. The same checkpoint can gain 5.5 points compacted and lose 3.8 points single-window, as CompactionRL's GLM-4.7-Flash does.
  • Comparing against a baseline with less effective context. SUPO's BrowseComp-Plus headline gives the trained model 3x the budget of the baseline. Match effective length or state that you have not.

References

  • ReSum: Unlocking Long-Horizon Search Intelligence via Context Summarization (ReSum-GRPO, ReSumTool-30B): https://arxiv.org/abs/2509.13313
  • Scaling LLM Multi-turn RL with End-to-end Summarization-based Context Management (SUPO): https://arxiv.org/abs/2510.06727
  • CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents: https://arxiv.org/abs/2607.05378
  • AutoCompact: Learning When to Compact Context in Long-Horizon Coding Agents (project page, no paper or code as of 2026-08-06): https://autocompact.github.io
  • MEM1 (constant-size internal state, the baseline both ReSum and SUPO compare against): https://arxiv.org/abs/2506.15841
  • MemAgent (segmented reading with an overwrite memory, subsumed by SUPO's framework): https://arxiv.org/abs/2507.02259
  • GAE (the advantage estimator the boundary correction modifies): https://arxiv.org/abs/1506.02438
  • DAPO (token-level loss and clip-higher, both reused here): https://arxiv.org/abs/2503.14476
  • BrowseComp-Plus (the shared evaluation environment): https://arxiv.org/abs/2508.06600
  • Terminal-Bench (Terminal-Bench 2.0 is the 89-task set used by CompactionRL): https://arxiv.org/abs/2601.11868

Related: When to compact · Agentic context management · Context and memory · GRPO variants · RL with GRPO · Async RL systems · Rollout reuse under policy lag · RL rollout redundancy · slime · Agent harness architecture · Hierarchical agent decomposition · Agent loop economics · Post-training system map