Markdown

TRACE: turn-level reward assignment via credit estimation¶

Scope: TRACE (Tao, Peng, Yao, Ge, Cheng, Wang, Gao, Li; Microsoft Research and University of Wisconsin-Madison, arXiv:2607.13988, July 2026), a dense credit-assignment method for long-horizon agentic RL that scores every tool-call boundary with a frozen reference model's gold-answer log-probability and turns the change in that score into a turn-level TD reward, mixed with the standard GRPO outcome advantage. This page covers the log-ratio value construction, the K-step backup with terminal-outcome fill, the BrowseComp-Plus results, and where the method's own stated limitations bound it. It sits next to reward design for RL post-training (the general shaping/normalization discipline TRACE is one instance of), agentic and tool-use RL (the ReAct rollout and loss-masking mechanics TRACE trains on top of), RLVR (the verifiable-outcome-reward paradigm TRACE keeps as its anchor), and RL scaling laws (how to read TRACE's faster-convergence learning curves against a compute budget).

Both numpy blocks below are independently written from the paper's equations and Algorithm 1, executed in this repository, and their asserts pass; one of them reproduces the paper's own Appendix A.2 worked-example numbers (1.1806 and 0.4052) to 1e-3 as a direct check against the source. Everything else (the datasets, model names, Table 1/2 scores, hyperparameters, and the Appendix A.5 trajectory excerpts) is quoted or derived from the PDF/HTML at arXiv:2607.13988, retrieved and read in full 2026-07-16. No code or checkpoint release is linked from the paper, the arXiv abstract page, or its HTML rendering as of that date; treat TRACE as a from-scratch reimplementation task, not a repo to clone.

What it is¶

TRACE (Turn-level Reward Assignment via Credit Estimation) is a critic-free way to give a long-horizon tool-using agent dense, per-turn reward instead of one reward for the whole rollout.¹ The problem it targets: an agentic RL rollout on a search task can run to tens or hundreds of tool calls before a verifiable final answer, and outcome-only RL (GRPO on right/wrong) assigns the same advantage to every turn in that rollout, so a failed trajectory's early useful searches get the same negative credit as the eventual wrong turn, and a successful trajectory's redundant searches get the same positive credit as the turn that actually found the answer.¹

TRACE's fix does not train a critic or a process reward model, and it does not need step-level human or judge labels. Instead it repurposes a frozen copy of the policy's own initialization as a stable probe: at each tool-call boundary (after each (action, observation) pair), it asks that frozen reference model how likely the gold answer is given everything the transcript has accumulated so far. A transcript that has gathered the right evidence should make the reference model more confident in the gold answer; a transcript that wandered should not.¹ Concretely, for prefix S_k (the prompt plus the first k action-observation pairs) the paper computes the reference model's average token log-probability on the gold answer y*:

l_bar_k = (1/|y*|) * sum_t log pi_ref(y*_t | S_k, y*_<t)   (<= 0, Eq. 5)

This raw log-probability is not directly usable as a per-turn reward: an identical absolute gain in l_bar_k means very different things depending on how close the trajectory already is to the answer (closing a gap from 0.2 to 0.1 removes half the remaining uncertainty; 5.1 to 5.0 removes almost none).¹ TRACE therefore defines the remaining gap d_k = -l_bar_k + eps (offset eps > 0 keeps it positive and finite) and a log-ratio state value

V(S_k) = log(d_0 / d_k)      (Eq. 6)

so V(S_0) = 0 and V(S_k) measures what fraction of the initial answer-likelihood gap has closed by turn k. The turn-level reward is the one-step TD change between adjacent values, delta_k = V(S_k+1) - V(S_k) = log(d_k / d_k+1) (Eq. 7): positive when the action and its returned observation make the gold answer more predictable, exactly log 2 when a transition halves the remaining gap, zero when nothing changes, negative when the transition moves away from the answer. Because these one-step credits telescope, sum_k delta_k = V(S_T) - V(S_0) = log(d_0/d_T), so the total credit over a trajectory depends only on the start and end states: padding a rollout with redundant tool calls cannot inflate it.¹

A pure one-step TD signal can still miss delayed effects (a browser.search that only surfaces candidate links, with the payoff arriving several turns later at a browser.open), so the method's reported runs use a K-step TD backup that spreads a discounted window of future one-step credits back onto the current turn, and anchors the last turns in that window to the verified outcome reward when the window reaches the trajectory's end (Eqs. 8-10, Algorithm 1). The resulting turn-level term is mixed, unnormalized, directly into the per-token advantage alongside the standard GRPO group-relative outcome advantage (Eq. 11), and the combined signal drives a standard clipped-surrogate GRPO update (Eq. 12).¹

Why use it¶

It substantially improves base-model tool use with pure RL. On the closed-web BrowseComp-Plus benchmark, TRACE raises Qwen3-4B-Thinking-2507 from a base score of 7.2 to 35.6, and Qwen3-30B-A3B-Thinking-2507 from 8.4 to 42.6, using only outcome-level and turn-level rewards, with no cold-start SFT stage, no agentic mid-training stage, and no live-web training data.³
It beats outcome-only GRPO under an otherwise identical setup. With backbone, browser interface, training data, terminal reward, and evaluation protocol held fixed, the four-benchmark average (BrowseComp-Plus, BrowseComp, GAIA, xbench-DeepSearch) rises from GRPO's 29.5 to TRACE's 34.0 on the 4B model, and from GRPO's 32.5 to TRACE's 38.1 on the 30B-A3B model. GSPO and GiGRPO, two other trajectory-level optimization variants trained under the same fixed setup, land at 28.2 and 26.5 (4B) and 33.3 and 29.7 (30B-A3B), below TRACE on both scales.³
The learned behavior transfers off the training corpus. Trained entirely on a closed synthetic-corpus search task, the 30B-A3B TRACE agent reaches 12.9 on the open-web BrowseComp benchmark, 52.0 on GAIA, and 45.0 on the Chinese-language xbench-DeepSearch, all served through an external search API rather than the training-time retrieval index.³
It converges faster, not just to a higher plateau. The paper's training-reward and held-out-accuracy curves (Figure 3) show TRACE's reward rising earlier and with a steeper early slope than GRPO's, and the gap persists rather than closing at convergence; at 160 steps the TRACE 30B-A3B checkpoint already exceeds GRPO's 200-step accuracy on BrowseComp-Plus.⁴ A companion trajectory-length plot (Figure 4) shows the same pattern in tool-call count: TRACE's average turns-per-rollout on the 4B model rises earlier and faster than GRPO's during early training, which the paper attributes to outcome-only GRPO being unable to separate "long trajectory that still failed" from "long trajectory that made progress," since both get the same negative group-relative advantage.⁴
No additional trained component and no strong judge. The reference model is a frozen copy of the policy's own initialization checkpoint, never optimized; the paper's own reference-checkpoint ablation (Figure 5c) found that swapping in a step-200 (partially trained) reference gave 36.1 versus the base checkpoint's 35.6, a difference the paper reads as evidence the method does not depend on a specially strong or carefully adapted reference model.⁵

Every number above is scoped to the paper's own controlled comparison: a single training run per configuration (the paper states ablations are single runs, so differences should be read as directional, not variance-adjusted), one closed synthetic training corpus, two Qwen3 backbones, and one search-agent task family.²

When to use it (and when not)¶

Use it for long-horizon, tool-heavy agentic RL where the final answer is short and verifiable. The paper's own scope statement is explicit: TRACE's value estimation works because the reference model scores the probability of a short gold-answer string, and this is a simple, stable proxy exactly when answers are compact and can be compared against ground truth.⁷
Use it when your terminal reward is already sparse and high-variance over many tool calls. TRACE's benefit shows up specifically on the benchmarks where success depends on a chain of dependent interactions (BrowseComp-Plus, xbench-DeepSearch); the paper attributes the gap over trajectory-level methods (GSPO, GiGRPO) to those methods still operating at the whole-rollout level even though they change how rollouts are compared to each other.³
Do not reach for it on tasks with long, structured, or open-ended outputs. The paper's stated limitation is direct: for a code agent producing a multi-file patch, or an open-ended assistant satisfying underspecified preferences, it is unclear whether gold-output log-probability under a frozen reference model is still a reliable state-value proxy; the authors explicitly leave alternative state-value targets (execution-based progress, decomposed verifiable subgoals) to future work.⁷
Do not treat the turn-level term as a replacement for the outcome reward. The paper's own coefficient sweep (Figure 5a) shows accuracy rising from 33.6 to 35.6 as the turn-reward weight increases from 1 to 3 (relative to the paper's units), then falling to 34.5 and 31.1 at larger weights, because an overweighted local reference-model signal starts to dominate final correctness rather than support it.⁵
Do not set the TD look-ahead horizon K to the trajectory length by default. The K-sweep (Figure 5b) shows accuracy at 30.0 with the dense backup disabled, rising to 34.7 and 35.6 at moderate K, then falling to 28.9 at the largest tested K, which the paper reads as the propagated signal picking up noise from loosely related later turns.⁵
Do not expect single-run ablation numbers to be statistically definitive. The paper says so itself: unless otherwise stated, its ablations are single training runs, and small differences should be read as directional.²

Architecture¶

flowchart TB
    Q["Prompt x, gold answer y*"] --> ROLL["Sample G rollouts from behavior policy pi_old
    ReAct loop: browser.search / browser.open / browser.find"]
    ROLL --> SPLIT["Split each rollout into prefixes S_0 .. S_T at tool-call boundaries"]
    SPLIT --> REF["Frozen reference model pi_ref
    scores gold-answer log-prob l_bar_k at every S_k (Eq. 5)"]
    REF --> VAL["Log-ratio state value
    V(S_k) = log(d_0 / d_k), d_k = -l_bar_k + eps (Eq. 6)"]
    VAL --> TD["One-step TD credit
    delta_k = V(S_k+1) - V(S_k) (Eq. 7)"]
    TD --> KSTEP["K-step discounted backup + terminal-outcome fill
    r_turn_k = c_k^(K) + 1[window reaches end] * lambda_term * gamma_td^(T-k) * A_out (Eqs. 8-10)"]
    ROLL --> OUT["Terminal verifier R(y_hat, y*)
    -> GRPO group-relative outcome advantage A_out (zero if group std = 0)"]
    KSTEP --> MIX["Per-token advantage
    A_hat = alpha_out * A_out + alpha_turn * r_turn (Eq. 11)"]
    OUT --> MIX
    MIX --> UPD["Clipped GRPO policy update J_TRACE (Eq. 12)
    tool-observation tokens masked from the loss"]
    UPD -->|"weight sync"| ROLL

The forward pass through the diagram runs once per rollout batch: sample trajectories, score every prefix with the frozen reference model in a single batched forward pass (the reference scores are never optimized), derive turn-level TD credit, compute the outcome advantage from the same batch's terminal rewards, mix the two per assistant token, and update the policy. Tool observations enter the transcript that the reference model scores but never receive a policy-gradient loss themselves; only assistant-generated action and answer tokens do.¹

How the log-ratio credit and K-step backup work (validated core math)¶

The two computations that make TRACE distinct from outcome-only GRPO are the log-ratio state value/TD credit (Eqs. 5-7) and the K-step backup with terminal-outcome fill (Eqs. 8-10, Algorithm 1). Both blocks below are self-contained numpy, executed in this repository, and the first one reproduces a worked numerical example directly from the paper's own Appendix A.2 as an assertion, not an illustration.

Log-ratio state value and one-step TD credit¶

The paper's Appendix A.2 argues that a raw log-probability difference and the log-ratio value give very different answers for two transitions with nearly the same absolute gain: l_bar going from -5.1187 to -1.5712 (raw gain 3.5475) versus from -10.6570 to -7.1061 (raw gain 3.5509), using the paper's diagnostic offset eps_diag = 1e-3. The paper reports log-ratio TD rewards of 1.1806 and 0.4052 for these two transitions respectively; the block below computes them from the definitions in Eqs. 6-7 and asserts they match to 1e-3.

# Runnable on system python3 (numpy). TRACE's log-ratio state value and one-step TD
# credit (arXiv:2607.13988, Eqs. 5-7): a frozen reference model's gold-answer log-prob
# at each tool-call boundary is turned into a relative-gap-closure value, and the TD
# difference between adjacent values is the turn's reward. Reproduces the paper's own
# Appendix A.2 worked example (two transitions with near-identical raw log-prob gains
# that the log-ratio form correctly separates) as a numerical assertion, not just a
# generic RL sanity check.
import numpy as np


def state_value(ell_bar: np.ndarray, eps: float) -> np.ndarray:
    """Eq. 6: V(S_k) = log(d_0 / d_k), d_k = -ell_bar_k + eps. V(S_0) = 0 by construction."""
    d = -ell_bar + eps
    assert np.all(d > 0), "gap d_k must stay positive; eps must exceed max(ell_bar)"
    return np.log(d[0] / d)


def td_credit(ell_bar: np.ndarray, eps: float) -> np.ndarray:
    """Eq. 7: delta_k = V(S_k+1) - V(S_k) = log(d_k / d_k+1) for k = 0..T-1."""
    d = -ell_bar + eps
    return np.log(d[:-1] / d[1:])


# --- 1. Telescoping: cumulative one-step credit depends only on the endpoints
# (paper, Section 3.3): a redundant middle transition (ell_bar unchanged) cannot
# inflate total credit, unlike a raw sum of positive per-step "progress" scores would.
eps = 0.1
ell_bar = np.array([-6.0, -4.0, -4.0, -4.0, -1.0])  # two no-op turns in the middle
V = state_value(ell_bar, eps)
delta = td_credit(ell_bar, eps)
assert V[0] == 0.0
assert abs(delta.sum() - (V[-1] - V[0])) < 1e-12, "telescoping identity violated"
assert abs(delta.sum() - np.log((-ell_bar[0] + eps) / (-ell_bar[-1] + eps))) < 1e-12

# ADVERSARIAL: redundant turns (no change in ell_bar, e.g. a repeated find that
# re-confirms an already-secured fact) must receive credit indistinguishable from
# zero, not a spuriously large or negative number.
assert abs(delta[1]) < 1e-12 and abs(delta[2]) < 1e-12, "redundant turn must score 0"

# --- 2. Harmful transition: a tool call that moves the trajectory away from the
# answer (ell_bar drops, i.e. the gold answer becomes LESS predictable) must get
# negative credit. This is the paper's Figure 1 / Appendix A.5 "negative turn" case.
ell_bar_bad = np.array([-3.0, -1.0, -1.0, -5.0])  # secured, then derailed
delta_bad = td_credit(ell_bar_bad, eps)
assert delta_bad[0] > 0, "evidence-gathering transition should get positive credit"
assert delta_bad[2] < -1.0, "derailing transition should get a sharp negative credit"

# ADVERSARIAL: near-certainty (ell_bar -> 0^-) must stay numerically finite because of
# the eps offset, never blow up to -inf/inf even as the remaining gap shrinks to eps.
ell_bar_confident = np.array([-8.0, -0.5, -1e-6])
delta_confident = td_credit(ell_bar_confident, eps)
assert np.all(np.isfinite(delta_confident)), "eps offset must prevent log(0)"

# --- 3. Reproduce the paper's own Appendix A.2 worked example verbatim: two
# transitions with almost the same raw log-probability gain (3.5475 vs 3.5509) get
# very different log-ratio TD rewards (paper: 1.1806 vs 0.4052) because the first
# closes a much larger fraction of its remaining gap. diagnostic offset eps=1e-3.
eps_diag = 1e-3
trans_a = np.array([-5.1187, -1.5712])   # Delta ell = 3.5475
trans_b = np.array([-10.6570, -7.1061])  # Delta ell = 3.5509
raw_a, raw_b = trans_a[1] - trans_a[0], trans_b[1] - trans_b[0]
credit_a = td_credit(trans_a, eps_diag)[0]
credit_b = td_credit(trans_b, eps_diag)[0]
assert abs(raw_a - raw_b) < 0.01, "raw deltas are near-identical, as the paper states"
assert abs(credit_a - 1.1806) < 1e-3, credit_a
assert abs(credit_b - 0.4052) < 1e-3, credit_b
assert credit_a > 2.5 * credit_b, "log-ratio must sharply separate near-equal raw gains"

print("TRACE state value / TD credit OK:",
      f"telescoping sum={delta.sum():.4f} == V_T-V_0={V[-1]-V[0]:.4f};",
      f"Appendix A.2 reproduction: credit_a={credit_a:.4f} (paper 1.1806),",
      f"credit_b={credit_b:.4f} (paper 0.4052)")

Executed output:

TRACE state value / TD credit OK: telescoping sum=1.7130 == V_T-V_0=1.7130; Appendix A.2 reproduction: credit_a=1.1806 (paper 1.1806), credit_b=0.4052 (paper 0.4052)

K-step backup with terminal-outcome fill¶

Algorithm 1 does not use the raw one-step delta_k as the training signal; it uses a K-step discounted average of the current and next K-1 one-step credits (Eq. 9), and additionally adds a discounted terminal-outcome term to any turn whose look-ahead window reaches the end of the trajectory (Eq. 10). The paper's own ablation note matters here: K=0 is a shorthand for disabling the dense backup entirely, not a value passed into Eq. 9 (which would divide by an empty window).¹ The block below implements Algorithm 1 lines 9-17 exactly, plus the GRPO zero-std outcome advantage from Algorithm 1 line 2.

# Runnable on system python3 (numpy). TRACE's K-step TD backup with terminal-outcome
# fill (arXiv:2607.13988, Algorithm 1 lines 9-17, Eqs. 8-10) and the GRPO zero-std
# outcome advantage it mixes with (Algorithm 1 line 2). This is the mechanism that
# lets a search/open pair get credit for evidence a LATER browser.open surfaces
# (delayed effects a pure one-step TD misses), while still anchoring the final few
# turns to verifiable outcome correctness.
import numpy as np


def group_relative_outcome_advantage(rewards: np.ndarray) -> np.ndarray:
    """Algorithm 1 line 2 / Section 3.3: standard GRPO group-relative advantage,
    returning 0 for every rollout in the group when the group std is 0."""
    r = np.asarray(rewards, dtype=np.float64)
    std = r.std(ddof=1) if len(r) > 1 else 0.0
    if std == 0.0:
        return np.zeros_like(r)
    return (r - r.mean()) / std


def k_step_turn_credit(
    delta: np.ndarray, K: int, gamma_td: float, lam_term: float, a_out_g: float
) -> np.ndarray:
    """Algorithm 1 lines 9-17 / Eqs. 8-10: turn-level reward for every transition
    k = 0..T-1 of one rollout with T tool interactions. delta are the one-step TD
    credits from Eq. 7 (len T). K=0 disables the dense backup entirely (paper's
    ablation note: K=0 is a shorthand for turning the term off, not an input to Eq. 9).
    """
    T = len(delta)
    r_turn = np.zeros(T)
    for k in range(T):
        if K == 0:
            c_k, h_k = 0.0, k
        else:
            h_k = min(k + K - 1, T - 1)
            u = np.arange(k, h_k + 1)
            weights = gamma_td ** (u - k)
            Z_k = weights.sum()
            c_k = (weights * delta[u]).sum() / Z_k
        terminal_fill = lam_term * gamma_td ** (T - k) if h_k == T - 1 else 0.0
        r_turn[k] = c_k + terminal_fill * a_out_g
    return r_turn


# --- 1. K=1 reduces exactly to the one-step TD credit (Zg,k has a single term,
# weight 1, so c^(K)_k = delta_k) whenever the window does not reach the trajectory
# end (no terminal fill contamination).
delta = np.array([0.8, -0.3, 1.5, 0.1, 2.0])   # T=5 transitions, mid-trajectory
r_k1 = k_step_turn_credit(delta, K=1, gamma_td=0.8, lam_term=0.0, a_out_g=0.0)
assert np.allclose(r_k1, delta), "K=1 with no terminal fill must equal one-step TD"

# --- 2. K=0 is the paper's explicit "disable the backup" shorthand: every turn's
# DENSE component (c^(K)) is exactly zero, distinct from passing K=0 through Eq. 9
# (which would be undefined, an empty window). With no terminal fill, K=0 zeroes
# the whole turn reward.
r_k0_no_fill = k_step_turn_credit(delta, K=0, gamma_td=0.8, lam_term=0.0, a_out_g=1.0)
assert np.all(r_k0_no_fill == 0.0), "K=0 must disable dense TD backup entirely (paper Sec 4.4)"

# But per Algorithm 1's own bookkeeping, K=0 sets h_g,k = k (line 12), so the
# terminal-fill indicator 1[h_g,k = T-1] still fires trivially for the LAST turn
# only (k = T-1). K=0 zeroes the dense backup everywhere; it does not remove the
# terminal outcome anchor from the final turn.
r_k0_with_fill = k_step_turn_credit(delta, K=0, gamma_td=0.8, lam_term=2.0, a_out_g=1.0)
assert np.all(r_k0_with_fill[:-1] == 0.0), "non-final turns must stay at 0 dense credit under K=0"
assert r_k0_with_fill[-1] != 0.0, "the last turn still receives the terminal anchor even at K=0"

# --- 3. Terminal-outcome fill only reaches turns whose look-ahead window touches
# the trajectory end (h_g,k == T-1), not every turn -- this is what makes Eq. 10
# "anchor the LAST turns", not the whole trajectory, to verifiable correctness.
K, gamma_td, lam_term, a_out = 3, 0.8, 2.0, 1.0
r_mixed = k_step_turn_credit(delta, K=K, gamma_td=gamma_td, lam_term=lam_term, a_out_g=a_out)
reaches_end = np.array([min(k + K - 1, len(delta) - 1) == len(delta) - 1 for k in range(len(delta))])
assert reaches_end.tolist() == [False, False, True, True, True], reaches_end
# turns 0,1 (window does not reach the end) get pure local TD progress, no fill
r_no_fill = k_step_turn_credit(delta, K=K, gamma_td=gamma_td, lam_term=0.0, a_out_g=0.0)
assert np.allclose(r_mixed[:2], r_no_fill[:2]), "turns whose window misses the end must be fill-free"
# turns 2,3,4 (window reaches the end) DO include the terminal fill term
assert not np.allclose(r_mixed[2:], r_no_fill[2:]), "turns reaching the end must include terminal fill"
assert np.all(r_mixed[2:] > r_no_fill[2:]), "positive a_out * lam_term must raise the filled turns"

# --- 4. ADVERSARIAL: a large-magnitude single outlier transition inside a wide
# window should be damped, not amplified, by the discounted average (Zg,k
# normalizes by the sum of weights, matching the paper's Fig 5(b) finding that an
# overly large K "introduces noise from loosely related later turns").
delta_outlier = np.array([0.1, 0.1, 50.0, 0.1, 0.1])  # one spurious spike at k=2
r_wide = k_step_turn_credit(delta_outlier, K=5, gamma_td=0.8, lam_term=0.0, a_out_g=0.0)
r_narrow = k_step_turn_credit(delta_outlier, K=1, gamma_td=0.8, lam_term=0.0, a_out_g=0.0)
assert r_wide[0] < r_narrow[2], "a distant spike must be damped, not fully passed through, at k=0"
assert r_wide[0] > 0, "the spike still leaks SOME positive signal backward under wide K"

# --- 5. ADVERSARIAL: single-turn trajectory (T=1, immediate answer after one tool
# call) with K larger than the trajectory. Must not index out of range; the window
# clamps to the only available transition and gets the terminal fill.
delta_short = np.array([1.2])
r_short = k_step_turn_credit(delta_short, K=10, gamma_td=0.8, lam_term=2.0, a_out_g=-1.0)
assert len(r_short) == 1
assert abs(r_short[0] - (1.2 + 2.0 * 0.8 ** 1 * -1.0)) < 1e-9

# --- 6. GRPO zero-std outcome advantage (Algorithm 1 line 2): a group where every
# rollout gets the same terminal reward (all correct or all wrong) must return
# advantage 0 for every member, never NaN/Inf, so it contributes no gradient signal
# through the outcome term (only the turn-level term, if any, still teaches).
rewards_flat = np.array([1.0, 1.0, 1.0, 1.0])
a_out_flat = group_relative_outcome_advantage(rewards_flat)
assert np.all(a_out_flat == 0.0) and np.all(np.isfinite(a_out_flat))

rewards_mixed = np.array([1.0, 0.0, 1.0, 0.0, 1.0, 0.0, 1.0, 0.0])
a_out_mixed = group_relative_outcome_advantage(rewards_mixed)
assert abs(a_out_mixed.mean()) < 1e-9
assert np.all((a_out_mixed > 0) == (rewards_mixed > 0.5))

print("TRACE K-step backup / terminal fill OK:",
      f"K=1 == one-step TD: {np.allclose(r_k1, delta)};",
      f"K=0 zeroed except last-turn anchor: {np.all(r_k0_with_fill[:-1] == 0.0)};",
      f"fill only reaches end-window turns: {not np.allclose(r_mixed[2:], r_no_fill[2:])};",
      f"outlier damped at k=0: {r_wide[0]:.4f} < narrow k=2: {r_narrow[2]:.1f};",
      f"zero-std group advantage: {a_out_flat.tolist()}")

Executed output:

TRACE K-step backup / terminal fill OK: K=1 == one-step TD: True; K=0 zeroed except last-turn anchor: True; fill only reaches end-window turns: True; outlier damped at k=0: 9.6002 < narrow k=2: 50.0; zero-std group advantage: [0.0, 0.0, 0.0, 0.0]

How to use it: implementing the reference-scoring pass¶

There is no released code, so using TRACE means implementing Algorithm 1 against your own rollout and training infrastructure. The paper's own experimental setup (Section 4.1, Appendix A.1 and A.3) fixes the pieces you need to reproduce:

Harness: a ReAct-style loop where each assistant turn contains private reasoning plus exactly one tool call or a final <answer> span. The paper's browser interface exposes three tools, browser.search(query, topn), browser.open(id, loc, num_lines), browser.find(pattern, cursor), over a FAISS index built with Qwen3-Embedding-8B on a closed offline corpus.²
Loss masking: policy-gradient loss applies only to assistant-generated action and answer tokens; tool observations are environment outputs and are masked out, exactly as in ordinary agentic RL loss masking (see agentic and tool-use RL).²
Reference scoring pass: for every sampled rollout, form the answer-ready prefixes S_0 .. S_T and run one batched forward pass of the frozen reference model to get l_bar_k at every prefix (Eq. 5). This reference model is never optimized; the paper's launch script keeps this path disabled by default and only turns it on for TRACE runs via a remote reference-model scoring endpoint, since it is a separate forward pass from the policy's own generation.⁶
Turn-credit computation: apply Eqs. 6-9 to get r_turn_g,k for every transition, then Eq. 11 to mix it with the outcome advantage into a per-token advantage, and optimize the clipped GRPO objective in Eq. 12.
Reported hyperparameters (Qwen3-30B-A3B-Thinking-2507 run, Tables 3-4): global batch size 128, 8 rollouts per prompt, up to 60 tool turns, learning rate 1e-6 constant, KL loss off, clip bounds (0.20, 0.28), training gap offset eps_train = 0.1, look-ahead horizon K = 3, TD discount gamma_td = 0.8, terminal-outcome fill enabled at scale 2.0, alpha_out = 1.0, alpha_turn = 0.2.⁶ Treat these as one paper's fitted operating point for one benchmark family, not universal defaults; the paper's own coefficient and K sweeps (Figure 5a-b) show both are sensitive and non-monotonic.

How to develop with it: extending or adapting the method¶

Swap the reference-scoring target for your domain, cautiously. The paper's answer-readiness proxy only works because the gold answer is a short, comparable string; the authors flag execution-based progress signals or decomposed verifiable subgoals as the natural extension for code or long-form tasks, but note this is unvalidated future work, not a drop-in substitution.⁷
Tune K and alpha_turn together, on your own task, before trusting the paper's values. Both ablations (Figure 5a-b) are non-monotonic with a moderate optimum; naively porting K=3, alpha_turn=0.2 to a different tool-call cadence or trajectory length distribution is not justified by anything in the paper.
The reference checkpoint does not need to be strong. The reference-model ablation (Figure 5c) found the base (step-0) checkpoint sufficient, with a step-200 checkpoint giving only a marginal +0.5 accuracy difference; this simplifies deployment (no separate "teacher" model to train or maintain) but was tested only within this paper's own training run, on one model family.⁵
Validate the ablation's isolated components separately before combining them. Table 2 shows each piece adds value in this run: outcome-only GRPO 30.0, plus raw log-probability delta 32.4, plus normalizing by the remaining gap (linear form) 34.6, plus the full log-ratio TD form 35.5. This is a single ablation run on one benchmark, and the paper reports it as such.⁵

How to run it in production¶

TRACE modifies the reward/advantage computation inside an existing agentic GRPO pipeline; it does not replace the rollout engine, the tool sandbox, or the weight-sync path. Two systems costs are specific to it:

A second frozen-model forward pass per rollout batch. Every sampled trajectory needs T+2 reference-model scoring calls (one before tool use, one after each observation, one after the answer, per Appendix A.5); at 8 rollouts per prompt and up to 60 tool turns, this is a non-trivial additional inference workload alongside the policy's own generation and the outcome verifier. The paper's launch script runs this scoring through a remote endpoint rather than colocating it with the trainer, keeping the reference-model compute off the training GPUs by default.⁶ Budget it as you would any additional inference-serving tier (see inference serving).
The reference model is a snapshot, not a moving target. Because it is frozen at (or near) initialization and never updated during the run, it does not need weight-sync coordination the way the rollout engine's policy copy does; treat it as a static served checkpoint, not part of the trainer's weight-sync loop.
Watch the same core RL health signals as any GRPO run (entropy, KL to reference, frac_reward_zero_std; see reward design and GRPO), plus the turn-level-specific ones: the coefficient sweep (Figure 5a) and K sweep (Figure 5b) both degrade past a moderate optimum, so a production run should track the turn-reward magnitude relative to the outcome term, not just the summed advantage.
Log per-turn credit for post-hoc debugging. The paper's own Appendix A.5 qualitative analysis (ten hand-inspected trajectories) shows the signal is directly interpretable: in every successful case examined, the large positive credit concentrates on the single search or open that supplied the decisive fact, and in every failed case, the trajectory reaches an answer-secured state (V >= -0.3 in the paper's diagnostic threshold) and then loses it to one sharply negative turn, a query-echo self-match, an over-interpreted empty find, or a match in the wrong context.⁸ This kind of per-turn trace is a debugging asset outcome-only training cannot produce; wire it into your training observability the same way you would any other dense diagnostic signal (see agent observability and the RL run-monitoring guidance in reward design).

Failure modes¶

Pitfall	Cause	Fix
Turn-level term drowns out final correctness	`alpha_turn` set too high; local reference-model readiness dominates the mixed advantage	Sweep the coefficient on your own task; the paper's own sweep peaks at a moderate value and degrades on both sides (Figure 5a)
Dense credit picks up noise from unrelated later turns	`K` set too large; the discounted backup window reaches loosely related future transitions	Sweep `K`; the paper's own sweep peaks at moderate K and drops sharply at the largest tested value (Figure 5b)
`K=0` silently produces near-GRPO results and looks like a bug	`K=0` is the paper's shorthand for disabling the dense backup entirely, not an input to Eq. 9	Confirm your implementation special-cases `K=0` as "dense term off," matching the paper's own ablation note
Turn credit blows up or divides by zero near a fully confident prefix	The remaining gap `d_k = -l_bar_k + eps` approaches zero without an offset	Keep `eps > 0` (paper uses `eps_train = 0.1` in training, `eps_diag = 1e-3` in offline diagnostics); never drop the offset
Reference-model scoring becomes the training bottleneck	An extra forward pass per prefix per rollout, at up to 60 tool turns and 8 rollouts per prompt, is a real inference cost	Serve the frozen reference model as its own tier (the paper routes it through a remote scoring endpoint) rather than colocating on trainer GPUs
Method applied to long or open-ended outputs with silently degraded signal	Gold-output log-probability under a frozen reference model is an unvalidated proxy outside short, compact, verifiable answers	Do not apply as-is; the paper explicitly scopes this out and leaves alternative state-value targets to future work
Single-run ablation numbers treated as statistically robust	The paper states its ablations are single training runs by default	Read differences as directional; re-run with multiple seeds before making a production decision on a specific coefficient or `K` value
Reported hyperparameters ported to a different tool-call cadence without retuning	Table 3/4 values are fit to one search-agent task at one trajectory-length distribution (up to 60 turns)	Re-tune `K`, `gamma_td`, `alpha_turn`, and the terminal scale for your own trajectory lengths and tool latencies

References¶

Tao, Peng, Yao, Ge, Cheng, Wang, Gao, Li, "TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents," arXiv:2607.13988 (2026-07-15): https://arxiv.org/abs/2607.13988, HTML: https://arxiv.org/html/2607.13988 (no code or checkpoint repository linked as of retrieval)
Chen, Ma, Zhuang, Nie, Zou, et al., "BrowseComp-Plus: A more fair and transparent evaluation benchmark of deep-research agent," arXiv:2508.06600: https://arxiv.org/abs/2508.06600
Wei, Sun, Papay, McKinney, Han, et al., "BrowseComp: A simple yet challenging benchmark for browsing agents," arXiv:2504.12516: https://arxiv.org/abs/2504.12516
Mialon, Fourrier, Swift, Wolf, LeCun, Scialom, "GAIA: A benchmark for general AI assistants," ICLR 2024: https://arxiv.org/abs/2311.12983
Chen et al., "xbench: Tracking agents productivity scaling with profession-aligned real-world evaluations," arXiv:2506.13651 (source of xbench-DeepSearch): https://arxiv.org/abs/2506.13651
Guo et al. (DeepSeek-AI), "DeepSeek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning" (GRPO), Nature 645 (2025): https://arxiv.org/abs/2501.12948
Yao, Zhao, Yu, Du, Shafran, Narasimhan, Cao, "ReAct: Synergizing reasoning and acting in language models," ICLR 2023: https://arxiv.org/abs/2210.03629
Sutton, "Learning to predict by the methods of temporal differences," Machine Learning 3 (1988) (the TD-error construction TRACE builds on): https://doi.org/10.1007/BF00115009
Yuan et al., "Free process rewards without process labels" (the closest prior implicit-reference-model credit method), arXiv:2412.01981: https://arxiv.org/abs/2412.01981
Zheng et al., "Group sequence policy optimization" (GSPO baseline), arXiv:2507.18071: https://arxiv.org/abs/2507.18071
Feng, Xue, Liu, An, "Group-in-group policy optimization for LLM agent training" (GiGRPO baseline), arXiv:2505.10978: https://arxiv.org/abs/2505.10978
Li et al., "OpenResearcher: A fully open pipeline for long-horizon deep research trajectory synthesis" (source corpus for TRACE's synthetic training data), arXiv:2603.20278: https://arxiv.org/abs/2603.20278

Tao et al., arXiv:2607.13988. Abstract and Section 1 (motivation, credit-assignment problem, headline BrowseComp-Plus numbers); Section 2.2 (TD preliminaries, Eq. 3); Section 3.1 (tool-boundary state notation, Eq. 4); Section 3.2 (reference answer score Eq. 5, log-ratio state value Eq. 6); Section 3.3 (one-step TD credit and telescoping Eq. 7, K-step backup and terminal fill Eqs. 8-10, mixed advantage Eq. 11, clipped objective Eq. 12); Algorithm 1 (full procedure, lines 1-22, including the K=0 special case at lines 11-12 and the zero-group-std outcome advantage at line 2). ↩↩↩↩↩↩↩↩
Tao et al., arXiv:2607.13988, Section 4.1 (Experimental Setup): training data built over the OpenResearcher offline corpus (Li et al., 2026); ReAct harness with browser.search/browser.open/browser.find tools, FAISS index with Qwen3-Embedding-8B; models Qwen3-4B-Thinking-2507 and Qwen3-30B-A3B-Thinking-2507, both starting from the base search policy with no cold-start SFT, mid-training, or live-web data; Adam, learning rate 1e-6, batch size 128, 8 rollouts/prompt, up to 60 tool turns; "unless otherwise stated, controlled ablations are single training runs, so small differences should be read as directional rather than as variance-adjusted conclusions." ↩↩↩↩
Tao et al., arXiv:2607.13988, Table 1 (main results: Qwen3-4B base 7.2 to TRACE 35.6 on BrowseComp-Plus, four-benchmark average GRPO 29.5 / GSPO 28.2 / GiGRPO 26.5 / TRACE 34.0; Qwen3-30B-A3B base 8.4 to TRACE 42.6, average GRPO 32.5 / GSPO 33.3 / GiGRPO 29.7 / TRACE 38.1; open-web transfer 30B-A3B TRACE 12.9 BrowseComp / 52.0 GAIA / 45.0 xbench-DeepSearch); Section 4.2 (Main Results narrative, including the caveat that TRACE "does not close the gap to the strongest external deep-research system in Table 1"). ↩↩↩↩
Tao et al., arXiv:2607.13988, Section 4.3 and Figure 3 (training reward and held-out accuracy curves on 4B and 30B-A3B, TRACE starting earlier, steeper early slope, higher plateau; 160-step TRACE 30B-A3B checkpoint exceeding 200-step GRPO on BrowseComp-Plus); Figure 4 (trajectory-scale / tool-call-count dynamics on 4B, TRACE rising earlier and faster than GRPO). ↩↩
Tao et al., arXiv:2607.13988, Section 4.4 (Ablation Study) and Table 2 (credit-format ablation: GRPO 30.0, +raw delta 32.4, +remaining gap (linear) 34.6, +log-ratio 35.5); Figure 5a (turn-level reward coefficient sweep, 33.6 to 35.6 to 34.5 to 31.1); Figure 5b (TD look-ahead horizon K sweep, K=0/off 30.0, K moderate 34.7/35.6, K largest 28.9); Figure 5c (reference-model ablation: no reference scoring 30.0, step-0 checkpoint 35.6, step-200 checkpoint 36.1). ↩↩↩↩↩
Tao et al., arXiv:2607.13988, Appendix A.1, Tables 3-4 (Qwen3-30B-A3B-Thinking-2507 Search-R1 run hyperparameters: SGLang rollout engine, global batch 128, 8 samples/prompt, temperature 1.0, max 60 tool turns, Adam lr 1e-6, clip bounds 0.20/0.28, KL loss 0; TRACE-specific: turn-reward weight 0.2, training gap offset eps_train=0.1, K=3, gamma_td=0.8, terminal-outcome fill enabled at scale 2.0, "turn-reward scoring enabled for TRACE runs with a remote reference-model scoring endpoint," disabled by default for a single-node colocated layout). ↩↩↩
Tao et al., arXiv:2607.13988, Section 6 (Limitations): empirical validation is scoped to long-horizon search agents with short, ground-truth-comparable final answers; the same value-estimation strategy "may be less direct" for long, structured, or open-ended outputs (multi-file code patches, underspecified user preferences); the authors explicitly leave alternative state-value estimators (execution-based progress, decomposed verifiable subgoals) to future work, and state the limitation "does not affect the main claim that turn-level credit can reduce the sparsity of outcome-only agentic RL" but does bound the method's scope. ↩↩↩
Tao et al., arXiv:2607.13988, Appendix A.5 (Qualitative Analysis of the Turn Credit): five successful and five failed trajectories from rollout batches 0140-0149; V >= -0.3 used as the diagnostic "answer-secured prefix" threshold; positive examples P1-P5 concentrate credit on the single decisive search/open turn; negative examples N1-N5 each reach an answer-secured prefix and then lose it to one sharply negative transition (query-echo self-match, over-interpreted empty find, match in wrong context, exact-string-vs-type mismatch, prompt-phrase-mistaken-for-page-text); closing summary: "Outcome-only GRPO cannot express any of these within-trajectory distinctions because it assigns every turn the same trajectory-level advantage." ↩