Markdown

Dream-RSI: replay simulators for recursive self-improvement of exploration¶

Scope: Dream-RSI (arXiv 2609.14858, Google, Google DeepMind, University of Maryland, University of Virginia), a meta-layer for agent-driven discovery loops that treats completed discovery trees as a replay simulator, evaluates candidate exploration policies on it at negligible cost, and redeploys the best one. Covers the decision interface, the replay score (Eq. 1), the loop, a runnable reconstruction of the replay core, reported results checked against the paper's tables, and the state of the public repository. Discovery loops themselves are in autonomous experimentation loops; this page is about optimising the exploration policy above them.

Reported numbers are quoted from the paper or recomputed from its tables. The code release is not public (see below), so nothing was reproduced. The Python block rebuilds the paper's replay mechanism on a synthetic world with illustrative constants.

What it is¶

An LLM coding agent proposes candidates, a fixed evaluator scores them, and an exploration policy decides which saved workspace each next attempt should start from, how many attempts run in parallel, and when to stop. Existing systems keep that policy fixed, or tune it online, which needs a whole discovery run per policy evaluated.

Dream-RSI's premise is that one finished run already stores a tree of outcomes. Formally:

Discovery tree. A rooted tree; each non-root node is one generation-and-evaluation attempt with a saved workspace and a score s_v (higher is better), and has exactly one parent.
Decision interface. The policy sees the revealed tree. Eligible nodes are the root plus current leaves. Each round it picks a batch C of at most W distinct eligible nodes, one attempt per node.
Online rollout. The attempts really run (stochastic); children attach to the tree; at most K1 rounds.
Replay. The policy runs over a stored tree. Selecting a non-root leaf reveals its single recorded child; selecting the root reveals the earliest-created branch not yet revealed. At most K2 rounds; it ends on an empty batch, the cap, or a fully revealed tree. No new outcomes are generated.
Replay score (Eq. 1). V = max_{v in revealed} s_v - beta1 * N + beta2 * N / max(1, k), where N is the number of revealed non-root nodes and k the rounds used. Quality, minus a per-attempt cost, plus a bonus for attempts per round (batching).
Loop. Deploy policy pi_t online to record tree T_t; add it to the history; a fixed policy-development LLM agent revises the policy code M times, each version scored as the mean of V over all recorded trees; deploy the argmax, which includes the incumbent, so the chosen version's replay score is at least the incumbent's.

Only exploration-policy code changes; the coding agent, evaluator, and execution interfaces are fixed.

Why use it¶

Cheap meta-feedback. A policy is evaluated by reading stored records, so many candidate policies can be scored per recorded run.
No weight updates. The paper reports zero gradient steps on the coding agent.
History used interactively. The paper's analysis (Figure 5, ConvDiv) reports that injecting distilled directional guidance into the prompt underperformed the unguided replay-based variant for both fixed and Dream-RSI policies.
Reported cost reduction. On Lasso path discovery, 317 discovery-agent calls against 550 for the fixed-exploration baseline with Gemini-3.1-Pro, and 1,879 against 3,200 with Gemini-3.7-Flash.

When to use it (and when not)¶

Use it when discovery runs are expensive, long (hundreds to thousands of attempts), and repeated, so that recorded trees accumulate, and when the search is tree-structured with resumable workspaces.

Do not expect it to help when:

there is only one or a few recorded runs: the replay world is the only evidence the policy is optimised against;
the quality of a new policy depends on regions never explored: replay returns only recorded outcomes, so a policy that would have found something different cannot be credited or blamed for it;
the cost that matters includes the policy-development agent and replay compute: the paper measures discovery cost as cumulative discovery-agent calls and does not report the cost of the policy-revision calls.

Architecture¶

flowchart LR
  P["Exploration policy pi_t (code)"] -->|"1 online explore"| T["Discovery tree T_t<br/>nodes: workspace, artifact, score"]
  T --> H["History H_t: pool of replay worlds"]
  H -->|"2 construct simulator"| S["Replay simulator<br/>reveals recorded children only"]
  D["Policy-development LLM agent"] -->|"3 revise policy code, M times"| C["Candidate policies"]
  C -->|"dream: score V on every tree"| S
  S -->|"replay trajectories and scores"| D
  S -->|"argmax mean V, incumbent included"| P

How to use it¶

The paper uses the same discovery setting for Dream-RSI and its main controlled baseline, Recursive Fixed Exploration, from the same hand-written starting policy (parallel independent workspaces, each refining its own candidate). Budgets per round: 10 workspaces by up to 11 refinement steps (110 calls) for Gemini-3.1-Pro, and 32 by 20 (640 calls) for Gemini-3.7-Flash, through the Gemini CLI. Dream-RSI keeps per-round budgets as caps but may use fewer attempts: the total 317 calls over 5 Lasso rounds is 63 per round on average, against 110 for the baseline.

The appendix of the paper reproduces the prompts for online exploration and for replay-based policy revision; the policy-revision prompt asks for an OptimalPolicy.solve(self, question, budget=None) implementation with one swept beta knob (see the objective discrepancy below). The repository does not contain code.

How to develop with it¶

Implementation hooks to build: a tree store that keeps per-node workspace, score, and diagnostics; a replay engine with the exact reveal rule above; a policy interface limited to the revealed prefix (the paper's prompts forbid policies from using best_so_far or budget counters to decide what to replay); and a policy-development agent loop.

The block below reconstructs the replay core on a synthetic world (branches are chains whose scores follow a branch-specific trend plus noise). It checks Eq. 1 against a hand-computed case, determinism, no look-ahead, legality of batches, the bound that replay cannot beat the recorded maximum, selection including the incumbent, and, adversarially, how the in-sample ranking behaves on fresh worlds. W, K2, beta1, beta2 and the world generator are assumptions; the paper does not give its coefficient values.

"""Replay-simulator core of Dream-RSI (arXiv 2609.14858, Section 3), reduced to a
synthetic branch-and-chain discovery world. Constants are illustrative, not from the paper."""
from __future__ import annotations
from typing import Callable
import numpy as np

W, K2, B1, B2 = 4, 12, 0.01, 0.005      # workers, replay round cap, beta_1, beta_2
ROOT = -1                               # the root node; an opened branch b is identified by b
Obs = dict[int, list[float]]            # revealed scores per opened branch, in order
Policy = Callable[[Obs], set[int]]


def make_world(rng: np.random.Generator, branches: int = 8, depth: int = 6) -> list[list[float]]:
    q = rng.normal(size=branches)
    d = np.arange(1, depth + 1)
    return [(q[b] * (1 - 0.8 ** d) + rng.normal(0, 0.1, depth)).tolist() for b in range(branches)]


def replay(world: list[list[float]], policy: Policy) -> tuple[float, int, int]:
    """Returns (best revealed score, N revealed nodes, k rounds). Root score is 0.0 by assumption."""
    obs: Obs = {}
    n = k = 0
    best = 0.0
    while k < K2:
        batch = policy({b: list(s) for b, s in obs.items()})
        legal = {ROOT} | {b for b, s in obs.items() if len(s) < len(world[b])}
        assert batch <= legal and len(batch) <= W, "illegal batch"
        if not batch:
            break
        for v in sorted(batch):
            b = len(obs) if v == ROOT else v        # root opens the earliest unrevealed branch
            if v == ROOT:
                if b == len(world):
                    continue
                obs[b] = []
            obs[b].append(world[b][len(obs[b])])
            n += 1
            best = max(best, obs[b][-1])
        k += 1
    return best, n, k


def score(world: list[list[float]], policy: Policy) -> float:
    best, n, k = replay(world, policy)
    return best - B1 * n + (B2 * n / max(1, k) if n else 0.0)       # Eq. (1)


def greedy_best(depth: int, open_n: int) -> Policy:
    """Open branches until open_n exist, extend the best-scoring live branches with spare workers."""
    def pi(obs: Obs) -> set[int]:
        opening = len(obs) < open_n
        live = sorted((b for b, s in obs.items() if len(s) < depth), key=lambda b: -obs[b][-1])
        return ({ROOT} if opening else set()) | set(live[: W - opening])
    return pi


def parallel_refine(depth: int) -> Policy:
    """Fixed baseline in the paper's spirit: open four branches, refine all of them to depth."""
    return greedy_best(depth, 4)


def memorized(fav: int) -> Policy:
    """Hard-codes branch index `fav`, the best branch of some recorded history."""
    def pi(obs: Obs) -> set[int]:
        if len(obs) <= fav:
            return {ROOT}
        return {fav} if len(obs[fav]) < 6 else set()
    return pi


# 1. Eq. (1) on a hand-checkable world.
tiny = [[0.5, 0.9], [0.2, 0.3]]
stop_now = lambda obs: set()
one_round = lambda obs: {ROOT} if not obs else set()
two_round = lambda obs: {ROOT} if not obs else ({0} if len(obs[0]) < 2 else set())
assert replay(tiny, stop_now) == (0.0, 0, 0) and score(tiny, stop_now) == 0.0
assert replay(tiny, one_round) == (0.5, 1, 1)
assert abs(score(tiny, one_round) - (0.5 - B1 + B2)) < 1e-12
assert replay(tiny, two_round) == (0.9, 2, 2) and abs(score(tiny, two_round) - (0.9 - 2 * B1 + B2)) < 1e-12
print("1 Eq.(1) matches hand computation")

# 2. Replay is deterministic, hides the future, and rejects illegal batches.
rng = np.random.default_rng(7)
w = make_world(rng)
assert score(w, parallel_refine(6)) == score(w, parallel_refine(6))
seen: list[int] = []
def spy(obs: Obs) -> set[int]:
    seen.append(max((len(s) for s in obs.values()), default=0))
    return {ROOT} if len(obs) < 2 else set()
replay(w, spy)
assert seen == [0, 1, 1], seen                        # only depth-1 nodes are visible after opening
for bad in ({5}, {ROOT, 0, 1, 2, 3}):
    try:
        replay(w, lambda obs, bad=bad: bad)
    except AssertionError:
        continue
    raise SystemExit("illegal batch accepted")
print("2 determinism, no look-ahead, batch legality enforced")

# 3. Replay only re-ranks recorded outcomes: its best never exceeds the recorded tree maximum.
for _ in range(200):
    wr = make_world(rng)
    assert replay(wr, greedy_best(6, 8))[0] <= max(max(s) for s in wr) + 1e-12
print("3 replay score bounded by recorded tree maximum")

# 4. Selection: argmax of mean replay score over history; the incumbent is a candidate, so V* >= V0.
history = [make_world(rng) for _ in range(5)]
fav_hist = [int(np.argmax([max(s) for s in h])) for h in history]
fav = max(set(fav_hist), key=fav_hist.count)
cands = {"incumbent": parallel_refine(6), "greedy": greedy_best(6, 8),
         "shallow": greedy_best(2, 8), "memorized": memorized(fav)}
V = {k: float(np.mean([score(h, p) for h in history])) for k, p in cands.items()}
pick = max(V, key=V.get)
assert V[pick] >= V["incumbent"]
print("4 in-sample V:", {k: round(v, 3) for k, v in V.items()}, "-> picks", pick)

# 5. The guarantee is in-sample only: a policy fitted to a fixed history can lose on fresh worlds.
fresh = [make_world(rng) for _ in range(300)]
out = {k: float(np.mean([score(f, p) for f in fresh])) for k, p in cands.items()}
print("5 fresh-world V:", {k: round(v, 3) for k, v in out.items()})
assert out["memorized"] < out["greedy"]
assert pick == "incumbent" and out["greedy"] > out["incumbent"]   # 5-world ranking inverts on 300 fresh worlds (seed 7)

Executed output (Python 3, numpy, seeded):

1 Eq.(1) matches hand computation
2 determinism, no look-ahead, batch legality enforced
3 replay score bounded by recorded tree maximum
4 in-sample V: {'incumbent': 0.441, 'greedy': 0.287, 'shallow': 0.233, 'memorized': 0.307} -> picks incumbent
5 fresh-world V: {'incumbent': 0.614, 'greedy': 0.678, 'shallow': 0.384, 'memorized': 0.285}

Step 4 and step 5 together are the point of the last check. The selection rule guarantees the chosen policy is not worse than the incumbent on the recorded worlds; here, with five worlds, it keeps the incumbent, and on 300 fresh worlds the rejected greedy policy scores higher (0.678 against 0.614). The ranking inverted because the selection set was small. This is one seed of a synthetic generator and shows the mechanism, not the magnitude in the paper's tasks.

How to maintain it¶

Grow the pool of trees and evaluate on a held-out subset of trees that policy revision never saw; the paper's rule selects on the same trees that feed the revision agent.
Pin the evaluator and the coding agent: changing either silently invalidates recorded scores as replay ground truth.
Report the policy-development and replay cost next to discovery calls when comparing with baselines.
Fix beta1 and beta2 per task family and record them; the quality-versus-cost-versus-batching trade in Eq. 1 is set by them, and the paper does not publish its values.

How to run it in production¶

Keep the deployed policy as reviewable code, versioned with the tree pool it was selected on.
Gate redeployment on replay improvement over the incumbent and, where a held-out tree exists, on that too.
Expect online behaviour to differ from replay: online children are stochastic, replay children are fixed.
Adaptive behaviour is reported rather than asserted: on ConvDiv (Figure 6, nine rounds E0 to E8), evaluated attempts per round fell from 110 to 50 while round-best performance rose, then went back up to the 80s and 90s once progress plateaued; round-best values in the figure text run from 0.427 to 1.898. The per-round mapping is not recoverable from the extracted figure text, so only these endpoints are quoted.

Reported results and inconsistencies¶

Lasso regularization path (six held-out datasets, average runtime in ms, lower is better; paper Figure 3a, table values):

System	Discovery calls	Avg runtime (ms)
sklearn	n/a	44,180.3
glmnet	n/a	13,767.5
SimpleTES (gpt-oss-120b)	51,200	3,804.8
SimpleTES marked with a dagger	51,200	8,318.4
Recursive Fixed Exploration, Gemini-3.1-Pro	550	3,587.1
Dream-RSI, Gemini-3.1-Pro	317	2,931.0
Recursive Fixed Exploration, Gemini-3.7-Flash	3,200	2,516.7
Dream-RSI, Gemini-3.7-Flash	1,879	2,350.6

Recomputed from the table: every row's average equals the mean of its six dataset columns. The ratios stated in the text hold: 3,587.1 / 2,931.0 = 1.22, 550 / 317 = 1.74, 3,200 / 1,879 = 1.70, and 51,200 / 317 = 161.5, reported as 162x.

Findings from the per-dataset columns that the headline hides:

The Gemini-3.1-Pro gain is one dataset. Dream-RSI is slower than Recursive Fixed Exploration on five of the six datasets (Gisette 2,841.0 against 1,861.8 ms; DNA 49.9 against 41.5; Leukemia 30.2 against 26.1; Colon 16.4 against 14.5; Duke Breast 32.5 against 28.4) and faster only on RCV1 (14,616.0 against 19,550.1). The RCV1 difference is 4,934 ms, larger than the whole six-dataset sum difference of 3,937 ms, so the other five datasets net about 998 ms slower. With Gemini-3.7-Flash, Dream-RSI wins five of six.
Against SimpleTES the average is also two large datasets. Both Dream-RSI rows lose to SimpleTES (first row) on DNA, Leukemia, Colon, and Duke Breast, often by 2x to 3x (DNA 49.9 against 15.9 ms), and win on Gisette and RCV1, which dominate the average. The "outperforms sklearn and glmnet on all six datasets" claim does hold for both Dream-RSI rows.
Undefined row. The table lists a second SimpleTES row marked with a dagger that the extracted paper text does not explain; the average is 2.2x the first row's. Which row is the like-for-like reference is not stated.
Units differ. "162x fewer calls" compares 317 Gemini-CLI discovery-agent calls with SimpleTES's 51,200 generations from a different model and system. It is a budget ratio, not a matched-compute comparison.

Mathematical optimization (10 rounds, Gemini-3.1-Pro; paper Table 1):

Task (direction)	Recursive Fixed	Dream-RSI	SimpleTES
Sum-Difference (higher)	1.144047	1.145427	1.143975
Autocorrelation (lower)	1.456001	1.456375	1.453675
Circle Packing (higher)	2.635983	2.635983	2.635983

Dream-RSI improves Sum-Difference by 0.001380 (0.12%), ties Circle Packing to six decimals, and is worse than its own fixed baseline on Autocorrelation (1.456375 against 1.456001; lower is better) and worse than SimpleTES (1.453675). The intro's "matches or surpasses strong baselines within 1k generations" therefore does not hold for Autocorrelation; the body says "competitive". No seeds, variances, or repeat counts are reported, so differences in the fifth decimal place cannot be separated from run-to-run noise.

KernelBench (four kernels, Gemini-3.1-Pro; paper Figure 4): 2.43x and 1.79x fewer generations to comparable performance on VGG16 and LayerNorm, and 2.09x and 1.44x higher performance at comparable budget on ConvDiv and ConvMax. The page reads these from figure annotations; absolute runtimes and per-kernel baselines are not tabulated, so the multipliers cannot be recomputed.

Other checks:

Repository statement versus paper. The repository README describes the arXiv posting as "coming soon" although arXiv 2609.14858 resolves; the README's "2 of 3 tasks at or above the selected baseline" matches Table 1 (Sum-Difference and Circle Packing).
Two different objectives. Eq. 1 in the body is a scalar with fixed coefficients beta1 and beta2. The appendix prompt given to the policy-development agent instead describes a policy with a single beta knob that the evaluator sweeps, ranked by pareto.reward = pareto.auc - lambda * parallel_penalty, where the penalty is the mean of effective sequential rounds over total probes. Both reward quality, punish probes, and favour batching, but they are not the same function, and the paper does not say which one selected the reported policies.
Guarantee scope. The paper's selection guarantee (V_{m*} >= V_0) holds on the fixed history. The paper presents no held-out-tree evaluation of the policies it selects.
No variance anywhere. Five Lasso rounds, ten math rounds, and the kernel curves are single trajectories as reported.

Failure modes¶

Overfitting to recorded trees. A policy revised against the same few trees can encode their idiosyncrasies (the synthetic check above reproduces the ranking inversion).
Gaming the score. Eq. 1 pays for batching (beta2 * N / k) independently of quality; a policy can raise V by issuing wide batches of low-value attempts when beta2 is large relative to beta1.
Stale worlds. Trees recorded under an older coding agent or evaluator no longer describe current outcomes.
Uncounted cost. Policy-development calls and replay runs are outside the reported discovery-call budget.
Counterfactual blindness. Replay cannot score a branch the online run never opened, so the policy space is restricted to re-ordering and re-batching recorded attempts.

Open questions and validation¶

Coefficients beta1, beta2, lambda, and the values of M, K1, K2, and W used per task are not stated in the extracted paper text.
The dagger on the second SimpleTES row is undefined in the text extracted from both the arXiv PDF and the repository's PDF.
The repository (read 2026-09-30, latest commit 4149ea9) holds a README, CITATION.cff, figures, and the paper PDF; the README lists discovered programs, the full codebase, and reproduction scripts as "being prepared". No official code was available to read or run.
Whether gains persist on held-out trees, across seeds, or when the policy-development cost is included is not reported.

References¶

Zheng, Wu, Zhang, He, Zhang, Coleman, Wei, Bai, and co-authors, "Dream-RSI: Recursive Self-Improvement through Evolving Worlds", arXiv 2609.14858: https://arxiv.org/abs/2609.14858
Project repository (paper PDF, README; code not yet released): https://github.com/zhengkid/Dream-RSI
Project page: https://www.dream-rsi.com
Ouyang et al., "KernelBench: Can LLMs write efficient GPU kernels?", arXiv 2502.10517: https://arxiv.org/abs/2502.10517
Novikov et al., "AlphaEvolve", arXiv 2506.13131: https://arxiv.org/abs/2506.13131
Hafner et al., "Mastering diverse domains through world models" (DreamerV3), arXiv 2301.04104: https://arxiv.org/abs/2301.04104
Ha and Schmidhuber, "World Models", arXiv 1803.10122: https://arxiv.org/abs/1803.10122