Markdown

Latent reasoning with outcome-reward RL¶

Scope: what it takes to run RLVR on a reasoner whose intermediate steps are continuous vectors rather than sampled tokens. This page covers the two things that are missing when you try (a tractable per-step likelihood and a variable stopping horizon), the surrogate policy that supplies the first, the hazard-rate stopping head that supplies the second, how the combined trajectory score plugs into RLOO or GRPO unchanged, and an honest read of what the published results do and do not show. It is the continuous-space counterpart to reasoning-effort control, which solves the same compute-allocation problem in token space, and it shares the outcome-reward machinery of RLVR and GRPO.

The numpy block below is executed and asserted in this page (Python 3.11, numpy 2.x). It implements the surrogate density, the gradient direction claimed by the paper's Prop. 1, the stopping-time distribution, and the variance-estimator bias, and it re-checks the paper's own Table 1 and cost profile for internal consistency. It does not train a latent reasoner. The accuracy figures are the paper's.

What it is¶

Latent reasoning carries intermediate computation as continuous vectors in hidden space instead of decoding each reasoning step as a natural-language token. The motivation is that much of a chain of thought exists for linguistic coherence rather than problem state, yet every one of those tokens pays the full cost of an autoregressive step. Latent reasoners (COCONUT, CODI, CoLaR, ReGuLaR and others) already match or surpass explicit chain of thought at far shorter horizons.

The problem SLPO addresses is that latent reasoners were stuck at imitation. Explicit chain of thought moved past teacher imitation years ago via RLVR, because each step is a token sampled from the vocabulary distribution, so there is a tractable per-step likelihood through which a trajectory-level reward can assign credit. A latent step has no such likelihood: the vector bypasses the vocabulary distribution entirely. Worse, existing latent reasoners prescribe a fixed thinking budget throughout training and inference, which freezes the exact compute horizon that RL would otherwise learn to allocate.

SLPO (Surrogate Latent Policy Optimization) supplies both missing pieces:

A surrogate transition likelihood. During training, run K stochastic forward evaluations with independent dropout masks on the same prefix. Take their empirical mean and a floored isotropic variance, and score the realised latent state under that Gaussian. This gives a differentiable per-step log-density where none existed.
A stopping head. A per-step stop probability (a discrete hazard rate) defines a distribution over stopping times. It is cold-started with correctness supervision, then refined by the same outcome reward, turning a fixed budget into a learned variable horizon.

The two are combined into one trajectory score, multiplied by a standard advantage, and handed to RLOO or GRPO with no other changes.

Why use it¶

Latency per unit of reasoning is the whole point of latent computation. A latent step does not decode a token. If reasoning is where your inference cost goes (reasoning-effort control exists because it usually is), moving that computation out of token space is a structurally different lever from making the tokens fewer.
It unlocks the optimisation stage the field already knows works. Imitation-bound models are capped by their teacher. Outcome-reward RL is what got explicit reasoners past that ceiling.
It converts a fixed budget into difficulty-adaptive compute. The released COCONUT and CODI checkpoints run a fixed T_max = 6 latent steps. SLPO keeps the same vector recurrence, raises the budget to T_max = 12, and lets the gate choose the realised length, spending more steps on harder instances.
It is algorithm-agnostic. The surrogate is an interface, not a new optimiser. The paper reports the effect persisting across both RLOO and GRPO, and transferring to soft-token latent inference.
The extra training cost is modest and controllable. Under the paper's configuration, surrogate construction is 24% of an optimiser step, and the cheaper knob (group size G) is the one that buys more.

When to use it (and when not)¶

Use it when you are already running a latent or continuous-CoT reasoner, you have a verifiable reward, and you have hit the imitation ceiling. This is a technique for people who have a latent reasoner, not a reason to acquire one.

Do not use it when:

You need greedy-decode accuracy. Read the results section carefully. The gains concentrate in Pass@k under parallel sampling; deterministic accuracy moves much less, and in one published cell it does not move at all.
You are working at frontier scale. The evidence base is GPT-2 (124M) and Llama-3.2-1B on grade-school math. Nothing here has been shown at the scale of the models in multi-teacher on-policy distillation.
Your reasoner is not autoregressive in hidden space. The surrogate scores latent transitions; a model without that recurrence has nothing to score.
You cannot run MC-dropout at training time. Dropout is the sampling mechanism here. A model trained or served without dropout gives you K identical forwards, a zero empirical variance, and a surrogate that is pinned at the variance floor.

Architecture¶

flowchart TB
  X["prompt x"] --> LR["latent recurrence f_theta"]
  subgraph ROLL["Rollout (MC-dropout, rate p=0.1)"]
    LR --> H["realised latent states h_1..h_tau<br/>(stop-gradient targets)"]
    H --> GATE["stopping head: rho_t"]
    GATE -->|"first t with rho_t > threshold"| TAU["stopping time tau"]
    TAU --> DEC["decode answer a"]
  end
  subgraph SUR["Surrogate construction (K forwards per step)"]
    LR --> Z["z^(1..K) with independent dropout masks"]
    Z --> MOM["mu_t, sigma^2_t (floored, 1/(K d))"]
    MOM --> LOGP["log pi_tilde(h_t) = isotropic Gaussian score"]
  end
  DEC --> R["verifiable reward R(a, a*)"]
  R --> ADV["advantage A = R - b<br/>(RLOO / GRPO baseline)"]
  LOGP --> SCORE["trajectory score:<br/>sum log pi_tilde(h_t)<br/>+ sum log pi(a_s)<br/>+ log P(tau)"]
  TAU --> SCORE
  ADV --> LOSS["L = -A * score"]
  SCORE --> LOSS
  LOSS -->|"gradients flow through mu, sigma^2 only"| LR

The direction of the gradient arrows is the subtle part. The realised rollout states are stop-gradient targets; gradients flow through the recomputed moments, not through the sampling path that produced the states.

How it works, validated¶

import numpy as np

# =========================================================== 1. the surrogate
def surrogate_logpdf(h, mu, sigma2, d=None):
    """Isotropic Gaussian surrogate over a latent transition (SLPO Sec. 4.2)."""
    d = len(h) if d is None else d
    return -0.5 * d * np.log(2 * np.pi * sigma2) - np.sum((h - mu) ** 2) / (2 * sigma2)


def mc_dropout_moments(samples, eps=1e-6):
    """Empirical mean and FLOORED isotropic variance over K stochastic forwards.
    Note the 1/(K*d) normalisation: this is the MLE, not Bessel-corrected."""
    K, d = samples.shape
    mu = samples.mean(axis=0)
    sigma2 = np.sum((samples - mu) ** 2) / (K * d) + eps
    return mu, sigma2


rng = np.random.default_rng(7)
d = 8
mu_t = rng.normal(size=d)
h = mu_t + rng.normal(size=d) * 0.3
s2 = 0.25

# It is a genuine density: matches the closed-form isotropic normal, and
# integrates to 1 (checked by quadrature in 1-D).
ref = -0.5 * d * np.log(2 * np.pi * s2) - np.sum((h - mu_t) ** 2) / (2 * s2)
assert np.isclose(surrogate_logpdf(h, mu_t, s2), ref)
grid = np.linspace(-40, 40, 400_001)
pdf1d = np.exp([surrogate_logpdf(np.array([g]), np.array([0.0]), s2) for g in grid])
assert abs(np.trapezoid(pdf1d, grid) - 1.0) < 1e-9

# ---- Prop. 1: a positive advantage pulls mu toward the realised state ------
def mu_step(mu, h, sigma2, adv, lr=0.05):
    """Ascent on adv * log pi(h); d/dmu = adv * (h - mu) / sigma2."""
    return mu + lr * adv * (h - mu) / sigma2


before = np.linalg.norm(mu_t - h)
assert np.linalg.norm(mu_step(mu_t, h, s2, adv=+1.0) - h) < before
assert np.linalg.norm(mu_step(mu_t, h, s2, adv=-1.0) - h) > before
assert np.allclose(mu_step(mu_t, h, s2, adv=0.0), mu_t)
assert surrogate_logpdf(h, mu_step(mu_t, h, s2, +1.0), s2) > surrogate_logpdf(h, mu_t, s2)
assert surrogate_logpdf(h, mu_step(mu_t, h, s2, -1.0), s2) < surrogate_logpdf(h, mu_t, s2)

# ---- the variance floor is load-bearing, not cosmetic ----------------------
collapsed = np.tile(rng.normal(size=d), (4, 1))          # all K forwards identical
mu_c, s2_c = mc_dropout_moments(collapsed)
assert s2_c == 1e-6 and np.isfinite(surrogate_logpdf(h, mu_c, s2_c))
mu_c0, s2_c0 = mc_dropout_moments(collapsed, eps=0.0)
assert s2_c0 == 0.0                                       # unfloored -> degenerate
with np.errstate(divide="ignore", invalid="ignore"):
    assert not np.isfinite(surrogate_logpdf(h, mu_c0, s2_c0))

# ---- the 1/(K*d) estimator is biased low by exactly (K-1)/K ---------------
TRUE_VAR, K, DIM, TRIALS = 0.5, 4, 64, 4000
est = np.empty(TRIALS)
for i in range(TRIALS):
    s = rng.normal(scale=np.sqrt(TRUE_VAR), size=(K, DIM))
    est[i] = mc_dropout_moments(s, eps=0.0)[1]
expected = TRUE_VAR * (K - 1) / K
assert abs(est.mean() - expected) < 0.01, (est.mean(), expected)
assert abs(expected / TRUE_VAR - 0.75) < 1e-12          # K=4 underestimates by 25%
big = np.array([mc_dropout_moments(rng.normal(scale=np.sqrt(TRUE_VAR), size=(16, DIM)),
                                   eps=0.0)[1] for _ in range(TRIALS)])
assert abs(big.mean() - TRUE_VAR * 15 / 16) < 0.01
assert abs(big.mean() - TRUE_VAR) < abs(est.mean() - TRUE_VAR)

# ======================================================= 2. the stopping head
def stop_time_pmf(rho, t_max=None):
    """P(tau = t) = rho_t * prod_{k<t} (1 - rho_k), with forced stop at T_max."""
    rho = np.asarray(rho, dtype=float)
    t_max = len(rho) if t_max is None else t_max
    surv, pmf = 1.0, np.zeros(t_max)
    for t in range(t_max):
        p = 1.0 if t == t_max - 1 else rho[t]
        pmf[t] = p * surv
        surv *= (1.0 - p)
    return pmf


rho = np.array([0.05, 0.10, 0.20, 0.30, 0.50, 0.60])
pmf = stop_time_pmf(rho)
assert np.isclose(pmf.sum(), 1.0)                        # proper distribution
assert np.all(pmf >= 0)
assert np.isclose(pmf[0], 0.05)
assert np.isclose(pmf[1], 0.10 * 0.95)
# The forced stop at T_max absorbs the remaining mass: without it the hazard
# chain leaks probability and the gate loss is scored against a sub-stochastic law.
leaky = rho * np.concatenate([[1.0], np.cumprod(1 - rho)[:-1]])
assert leaky.sum() < 1.0 and abs(leaky.sum() - 0.90424) < 1e-5


def first_stop(rho, threshold):
    """Inference rule: stop at the first step whose stop probability exceeds
    the threshold; otherwise run to the budget."""
    idx = np.flatnonzero(np.asarray(rho) > threshold)
    return int(idx[0]) + 1 if idx.size else len(rho)


assert first_stop(rho, 0.25) == 4
assert first_stop(rho, 0.99) == len(rho)                 # never fires -> full budget
assert first_stop(rho, 0.0) == 1                         # fires immediately

# ---- difficulty-adaptive computation falls out of the hazard ---------------
easy = np.full(12, 0.45)
hard = np.full(12, 0.12)
e_easy = float((np.arange(1, 13) * stop_time_pmf(easy)).sum())
e_hard = float((np.arange(1, 13) * stop_time_pmf(hard)).sum())
assert e_hard > e_easy
assert e_easy < 3.0 < e_hard
# Raising the budget from T_max=6 to T_max=12 (what +SLPO does to the released
# checkpoints) is nearly a no-op for easy instances and a large change for hard
# ones: the extra budget is spent only where the hazard is low enough to reach it.
e6_easy = float((np.arange(1, 7) * stop_time_pmf(easy[:6])).sum())
e6_hard = float((np.arange(1, 7) * stop_time_pmf(hard[:6])).sum())
assert e6_easy / e_easy > 0.95, e6_easy / e_easy
assert e6_hard / e_hard < 0.85, e6_hard / e_hard

# =============================================== 3. the published result table
# Table 1 of arXiv:2607.19691v2. (Acc, P@8, P@16) per dataset.
T1 = {
    ("GPT2", "COCONUT"):  {"GSM8K": (34.12, 45.79, 49.13), "GSM-Hard": (7.66, 10.70, 12.06),
                           "MultiArith": (80.86, 88.79, 91.38)},
    ("GPT2", "COCONUT+"): {"GSM8K": (35.63, 49.13, 51.55), "GSM-Hard": (7.66, 10.85, 12.52),
                           "MultiArith": (83.10, 91.38, 92.59)},
    ("GPT2", "CODI"):     {"GSM8K": (42.30, 52.08, 54.36), "GSM-Hard": (9.26, 11.84, 12.67),
                           "MultiArith": (90.21, 96.90, 97.41)},
    ("GPT2", "CODI+"):    {"GSM8K": (42.76, 54.13, 56.71), "GSM-Hard": (9.71, 12.06, 13.05),
                           "MultiArith": (90.52, 97.24, 97.76)},
    ("L1B", "COCONUT"):   {"GSM8K": (21.23, 24.64, 38.13), "GSM-Hard": (4.55, 6.98, 9.94),
                           "MultiArith": (41.10, 45.00, 63.10)},
    ("L1B", "COCONUT+"):  {"GSM8K": (22.37, 30.48, 41.85), "GSM-Hard": (5.77, 8.19, 10.93),
                           "MultiArith": (46.90, 57.07, 71.38)},
    ("L1B", "CODI"):      {"GSM8K": (55.22, 63.91, 67.48), "GSM-Hard": (12.82, 15.02, 15.63),
                           "MultiArith": (95.52, 98.28, 98.79)},
    ("L1B", "CODI+"):     {"GSM8K": (55.27, 65.13, 70.28), "GSM-Hard": (13.20, 15.86, 16.77),
                           "MultiArith": (96.38, 98.79, 99.48)},
}
PAIRS = [(("GPT2", "COCONUT"), ("GPT2", "COCONUT+")), (("GPT2", "CODI"), ("GPT2", "CODI+")),
         (("L1B", "COCONUT"), ("L1B", "COCONUT+")), (("L1B", "CODI"), ("L1B", "CODI+"))]
DATASETS = ["GSM8K", "GSM-Hard", "MultiArith"]

settings, acc_d, p8_d, p16_d = 0, [], [], []
for base, slpo in PAIRS:
    for ds in DATASETS:
        b, s = T1[base][ds], T1[slpo][ds]
        settings += 1
        assert s[1] > b[1], (base, ds, "Pass@8")      # improves in EVERY setting
        assert s[2] > b[2], (base, ds, "Pass@16")
        acc_d.append(s[0] - b[0]); p8_d.append(s[1] - b[1]); p16_d.append(s[2] - b[2])

assert settings == 12                                  # 2 reasoners x 2 backbones x 3 sets
assert abs(max(p8_d) - 12.07) < 1e-9                   # the headline gain
assert np.argmax(p8_d) == DATASETS.index("MultiArith") + 3 * 2

# The honest caveat: deterministic accuracy barely moves, and one cell does not
# move at all. The method buys parallel-sampling coverage, not greedy accuracy.
assert min(acc_d) == 0.0
# Ordering: greedy accuracy moves least, and Pass@8 moves MORE than Pass@16,
# because the k=16 baseline has already covered much of the reachable headroom.
assert np.mean(acc_d) < np.mean(p16_d) < np.mean(p8_d)
assert np.mean(acc_d) < 1.3 and np.mean(p8_d) > 2.5

# ================================================ 4. the training cost profile
STEP = {"total": 1.130, "rollout": 0.294, "surrogate": 0.274, "backward": 0.545}
parts = STEP["rollout"] + STEP["surrogate"] + STEP["backward"]
assert parts <= STEP["total"] and parts / STEP["total"] > 0.98
assert abs(STEP["surrogate"] / STEP["total"] - 0.24) < 0.005
assert abs(0.676 / STEP["total"] - 0.60) < 0.005        # G: 8 -> 4
assert abs(2.021 / STEP["total"] - 1.79) < 0.005        # K: 4 -> 8
# Halving G saves 40% of a step; doubling K costs 79% more. Spend on G.
g_saving = 1 - 0.676 / STEP["total"]
k_cost = 2.021 / STEP["total"] - 1
assert k_cost > g_saving

print("all SLPO assertions passed")
print("  settings improved on Pass@8 and Pass@16:", settings, "/ 12")
print("  max Pass@8 gain (pp):", round(max(p8_d), 2))
print("  mean deltas  Acc / P@8 / P@16 (pp):",
      round(float(np.mean(acc_d)), 2), "/", round(float(np.mean(p8_d)), 2),
      "/", round(float(np.mean(p16_d)), 2))
print("  K=4 variance estimator bias factor:", round(expected / TRUE_VAR, 3))
print("  E[tau] easy / hard (T_max=12):", round(e_easy, 2), "/", round(e_hard, 2))

Executed output:

all SLPO assertions passed
  settings improved on Pass@8 and Pass@16: 12 / 12
  max Pass@8 gain (pp): 12.07
  mean deltas  Acc / P@8 / P@16 (pp): 1.2 / 2.53 / 2.07
  K=4 variance estimator bias factor: 0.75
  E[tau] easy / hard (T_max=12): 2.22 / 6.54

The surrogate is a real density, and its gradient does the obvious thing¶

Block 1 confirms the surrogate is a proper isotropic Gaussian log-density (it integrates to 1) rather than an unnormalised score, which matters because the trajectory score is summed with genuine log-likelihood terms for the answer tokens and the stopping time. Mixing a normalised term with an unnormalised one would silently reweight the three contributions.

The Prop. 1 check makes the learning signal concrete. Differentiating A * log pi_tilde(h) with respect to the mean gives A * (h - mu) / sigma^2. With a positive advantage the mean moves toward the realised latent state, making that transition more likely next time; with a negative advantage it is pushed away. That is exactly the behaviour you want from a policy gradient, and the block asserts both the distance and the log-density move in the right direction.

Note the 1/sigma^2 factor: the step size is inversely proportional to the surrogate variance. Steps where MC-dropout produced tightly clustered forwards get large updates, and steps where the model is genuinely uncertain get small ones. That is defensible, but it means the variance floor epsilon is also an implicit maximum learning rate.

Two things about the variance estimator that the equations do not advertise¶

The floor is load-bearing. If all K dropout forwards coincide, the empirical variance is exactly zero and the log-density is not finite. Block 1 constructs that case. Without the floor a single collapsed step poisons the entire batch's loss. This is not hypothetical: it is what you get if dropout is disabled, if the dropout rate is very low, or if a layer is in eval mode by accident.

The estimator is biased low by exactly (K-1)/K. The published normalisation is 1/(K*d), which is the maximum-likelihood variance rather than the Bessel-corrected one. At the paper's K = 4 that is a 25% underestimate, verified by Monte Carlo in the block. Combined with the 1/sigma^2 gradient scaling, a systematically small variance means systematically large steps. This is a defensible choice (it is consistent, and the floor bounds the worst case), but if you re-implement SLPO and your effective learning rate feels wrong by a constant factor, this is where to look.

The stopping head is a discrete hazard model¶

P(tau = t) = rho_t * prod_{k<t} (1 - rho_k) is a standard discrete-time hazard. Block 2 checks the consequence that is easy to get wrong: the chain only forms a proper probability distribution if the last step is a forced stop. Without it the hazard sequence in the example leaks about 9.6% of its mass, and the gate's log-likelihood is being computed against a sub-stochastic law. Cold-start supervision trains the gate to place mass on stopping times in the correctness-validated set; the outcome reward then reshapes it, because higher-reward stopping times reinforce their continue-or-stop decisions through the log P(tau) term in the trajectory score.

At inference the rule is simply the first step whose stop probability exceeds a threshold, with no enumeration of candidate prefixes. Block 2 checks the boundary behaviour: a threshold above every hazard runs to the full budget, and a threshold of zero stops immediately. That threshold is a serving-time knob, and it is the closest thing latent reasoning has to the effort selector described in reasoning-effort control.

The difficulty-adaptive result also falls straight out of the model. A low hazard means a long expected trajectory; the block shows expected length of 2.22 steps against 6.54 for the same budget. It also shows why raising T_max from 6 to 12 is nearly a no-op on easy instances (their expected length barely changes) but a large change on hard ones. If your traffic is mostly easy, raising the budget buys almost nothing.

How to read the results honestly¶

The headline is real: Pass@8 and Pass@16 improve in all 12 backbone-by-dataset settings, with a maximum gain of 12.07 points (Llama-3.2-1B COCONUT on MultiArith, Pass@8 from 45.00 to 57.07). Block 3 checks every one of those 24 comparisons rather than taking the claim on faith.

Three qualifications belong next to it:

The gains are in Pass@k, not in greedy accuracy. Mean deterministic-accuracy delta across the 12 settings is 1.20 points, against 2.53 for Pass@8. One cell (GPT-2 COCONUT on GSM-Hard) shows an accuracy delta of exactly 0.00. If you deploy with a single sample, most of this benefit is not available to you. The block asserts this ordering explicitly so the caveat cannot drift out of the page.
Pass@8 gains exceed Pass@16 gains (2.53 against 2.07 on average). That is the expected shape: at k=16 the baseline has already covered more of the reachable answer set, so there is less headroom. It also means the advantage narrows as you sample more, which is worth knowing before you build a Best-of-N deployment around it.
The scale is small and the domain is narrow. GPT-2 at 124M and Llama-3.2-1B, trained on GSM8K-Aug, evaluated on GSM8K-Test, GSM-Hard and MultiArith. Absolute GSM-Hard numbers are in the 5 to 17% range. This is a method paper demonstrating that an optimisation interface works, not evidence that latent reasoning is production-ready at scale.

How to develop with it: hyperparameters and cost¶

The paper's configuration: RLOO advantages over G = 8 sampled latent trajectories, K = 4 MC-dropout forwards per surrogate, dropout p = 0.1, Adafactor at learning rate 1e-6, up to 40,000 steps with 2,000 warmup, per-device batch 16 across 4 GPUs. Notably no PPO-style clipping and no KL penalty: beta = 0 and no reference-policy log-probability terms. That is a meaningful simplification relative to the usual RLVR setup and one to keep in mind if you port it into a trainer that assumes a reference model.

Block 4 checks the published per-step profile for internal consistency and turns it into a spending rule. At (K, G) = (4, 8) a step is 1.130 s, of which surrogate construction is 0.274 s (24%), rollout 0.294 s, and backward 0.545 s. The two available moves are asymmetric:

Change	Step time	Relative
baseline `(K, G) = (4, 8)`	1.130 s	1.00x
`G: 8 -> 4`	0.676 s	0.60x
`K: 4 -> 8`	2.021 s	1.79x

Doubling K costs 79% more per step; halving G saves 40%. Since the paper's own hyperparameter analysis finds larger returns from rollout diversity than from additional surrogate samples, the rule is: spend on G, keep K moderate. Four forwards is enough to build a usable surrogate even though it is a poor variance estimator, which is consistent with the observation that the surrogate only has to point the gradient in the right direction.

How to run it in production¶

Almost nothing here is a serving-time change. The trained artefacts are the same latent recurrence plus a stopping head, and the surrogate exists only during training. Two things do carry over:

The stop threshold is your effort knob. Lower it for cheaper, shorter latent trajectories; raise it to let the gate run longer. Unlike a token budget, it is dimensionless and does not need per-prompt estimation.
T_max still bounds worst-case latency. The gate is a soft policy; the budget is the hard cap. Keep it, and set it above the length the gate actually uses, or you will clip the hard instances that the adaptive horizon exists to serve.

If you are serving with dropout disabled (you should be), the deterministic-accuracy caveat above is the number that applies to you, not the Pass@k figure.

Failure modes¶

Dropout off, or too weak. K identical forwards give zero empirical variance, the floor takes over, and every step gets the same maximum-magnitude gradient. Symptom: surrogate variance pinned at epsilon across the batch.
Missing variance floor. Non-finite loss from a single collapsed step.
Hazard chain without a forced terminal stop. The stopping distribution sums to less than 1 and the gate term is mis-scored. Easy to miss because nothing crashes.
Cold start skipped. The gate is refined by the outcome reward, not created by it. Without correctness-supervised initialisation there is no sensible stopping policy for RL to improve.
Gradients flowing through the rollout sampling path. The realised states are stop-gradient targets by design; letting gradients through them changes the objective into something the paper does not analyse.
Raising T_max and expecting uniform benefit. The extra budget is only reachable where the hazard is low, so easy instances ignore it.
Reading Pass@k as if it were accuracy. The most likely way to be disappointed by this method in practice.
Assuming the surrogate gradient is the true reward gradient. It is not. The update is the gradient of an explicitly defined empirical surrogate objective, and the paper gives only a sufficient condition under which it locally improves the true expected reward. Treat divergence as possible rather than excluded.

References¶

SLPO: Scaling Latent Reasoning via a Surrogate Policy: https://arxiv.org/abs/2607.19691
SLPO project page and code: https://github.com/ModalityDance/SLPO
COCONUT (Chain of Continuous Thought): https://arxiv.org/abs/2412.06769
CODI (self-distilling CoT into continuous space): https://arxiv.org/abs/2502.21074
RLOO (leave-one-out baseline for LLM RL): https://arxiv.org/abs/2402.14740
GSM-Hard and the PAL benchmark family: https://arxiv.org/abs/2211.10435