Markdown

Frontis-MA1 and OpenMLE: training the improver, not just the solution¶

Scope: the OpenMLE stack and the Frontis-MA1 models from arXiv 2607.28568 (Frontis.AI and Tsinghua, submitted 2026-07-30), covering the three layers the paper releases (OpenMLE-Gym task environments, OpenMLE-ERL operator training, OpenMLE-Evo long-horizon search), the reward-shaping mathematics that makes execution feedback trainable, the cluster topology needed to reproduce any of it, and a careful separation of what the headline numbers do and do not establish. This is the "train the improver" counterpart to the frozen-model scaffolds in self-improving agent harnesses and autonomous experimentation loops; it is evaluated on MLE-Bench Lite and on NatureBench Lite, and its RL layer is a concrete instance of the reward-shaping problems in reward design for RL post-training.

Sourcing and naming notes, verified 2026-08-01. The arXiv abstract page names the middle layer OpenMLE-RL; the PDF, the repository directory, and every config call it OpenMLE-ERL. Both refer to the same component; the repository README uses the two spellings in adjacent tables. All numbers below come from the v1 PDF (arXiv:2607.28568v1, 61 pages) and were cross-checked against docs/results.md in the released repository, which reproduces the same tables. The Python block is executed and asserted (numpy only), and its outputs were additionally cross-checked against the shipped OpenMLE-ERL/RL/adaptive_reward_advantage_utils.py at commit c16ebd6; agreement was exact to 1e-9 over 500 random reward groups. Model names in the comparison tables are 2026-era and will date quickly.

flowchart TB
  subgraph GYM["OpenMLE-Gym (environment layer)"]
    SRC["Curated anchors 156<br/>Kaggle datasets 3,362<br/>Kaggle competitions 2,240"] --> PKG["Task package<br/>data/public + data/private<br/>utils/prepare.py + utils/metric.py"]
    PKG --> SB["Sandbox job service<br/>POST /api/v1/jobs, poll to terminal state<br/>NOT in the public repo"]
  end
  SB -->|"score, logs, error class, artifacts"| ERL
  subgraph ERL["OpenMLE-ERL (learning layer)"]
    SFT["Execution-grounded SFT<br/>26,259 examples<br/>17,245 Draft + 9,014 trajectory steps"] --> RL["GSPO + adaptive bounds<br/>+ entropic advantage<br/>16 prompts x 16 samples"]
  end
  RL --> MODEL["Frontis-MA1-35B / 30B<br/>Draft | Improve | Debug | Crossover"]
  MODEL --> EVO
  subgraph EVO["OpenMLE-Evo (search layer)"]
    SEL["Parent selection<br/>quality + progress + novelty"] --> MEM["Operator-conditioned memory<br/>synthesised on demand"]
    MEM --> EXP["Expand, execute, score"]
    EXP -->|"experience card"| SEL
  end
  EVO -->|"verified transitions become training data"| SFT
  EVO --> OUT["MLE-Bench Lite 22 tasks<br/>12h sandbox budget per task<br/>1x RTX 4090 capped at 12 GB"]

What it is¶

OpenMLE is a three-layer open stack for machine learning engineering agents, and Frontis-MA1 is the model it produces. The organising idea is a shared action space: instead of training on whole agent trajectories, the authors define four atomic program-transformation operators (Draft writes a program from scratch, Improve refines a scoring parent, Debug repairs a failing one, Crossover recombines two parents), train each operator directly from sandbox execution outcomes, and then let an evolutionary search harness compose those same operators at inference time.¹ The paper calls the resulting loop meta-evolution: search produces verified transitions, those transitions supervise the operators, and the improved operators drive the next search.

The three layers are separable and are released separately. OpenMLE-Gym turns Kaggle competitions and datasets into 5,758 quality-gated executable task packages, each with agent-visible inputs under data/public/, hidden answers under data/private/, and a task-specific metric.py that returns a scalar.² OpenMLE-ERL runs execution-grounded SFT on 26,259 examples followed by online RL with GSPO, adaptive reward bounds, and an entropic advantage.³ OpenMLE-Evo is an AIRA-Evo-derived population search that adds structured experience records, three-factor parent selection, and lazily synthesised operator-conditioned memory.⁴

Two checkpoints ship: Frontis-MA1-35B (35.95B parameters BF16, post-trained from Qwen3.6-35B-A3B, a multimodal MoE) and Frontis-MA1-30B (30.53B, from Qwen3-30B-A3B-Thinking-2507, 128 experts with 8 active).⁵ Both are MoE models with roughly 3B active parameters per token, which is what makes a 35B-class agent affordable to serve for hundreds of search steps per task. See MoE sparsity and scaling for why the active-parameter count, not the total, sets the serving cost.

Why use it¶

It closes the loop that most agent work leaves open. Inference-time scaffolds such as AIDE and AIRA amplify a frozen backbone; post-training work teaches a model to solve tasks but rarely exposes the same interface the scaffold calls. OpenMLE trains Improve, Debug, and Crossover as first-class targets, so the harness composes transformations the model was explicitly optimised for rather than prompting a general coder into them.
The controlled comparisons are genuinely controlled. Under one fixed harness, swapping only the model moves Medal Average from 39.39% to 60.61% on the 35B pair and from 34.85% to 53.03% on the 30B pair.⁶ Both gaps are large relative to the reported run-to-run dispersion: the 35B gap is about 3.8 standard errors of the difference across three runs, the 30B gap about 6.6.⁷ That is a cleaner attribution of credit than most agent papers offer.
The reward machinery is fully specified and it matches the code. Appendix B.5 gives the adaptive-bound construction and the entropic advantage; the four shipped launch configs set exactly the constants the appendix describes (ADAPTIVE_REWARD_BOUND_MODE=top1_top16, ADAPTIVE_REWARD_BOUND_LOWER_SHIFT_RATIO=0.25), and ttt_reward_postprocess.py hard-codes the log 2 KL target, 60 bisection iterations, and 1e6 beta ceiling that the appendix states.⁹ Paper-to-code agreement at this level of detail is rare enough to be a reason to trust the rest.
It is the most complete public artifact set in its category, by the authors' own audit. Their Table 11 scores fourteen comparable systems on six release surfaces (data, sandbox, training code, RL method, eval, weights); OpenMLE is the only row with all six.¹⁰ For a team trying to stand up an MLE-agent research loop rather than consume one, that matters more than the leaderboard position.
The search efficiency work is directly reusable. Against original AIRA-Evo on the same checkpoint and seed, operator-conditioned bounded context cuts total model tokens 41.7% and the 99th-percentile Improve prompt from 389.0K to 54.3K characters.¹¹ Those are context-engineering wins that transfer to any long-horizon agent, independent of MLE.

When to use it (and when not)¶

Use OpenMLE-Gym when you need thousands of executable, scored ML tasks for post-training or search research and the 75-task MLE-Bench is too small a substrate. The task-package contract (public inputs, private answers, executable metric) is worth copying even if you build your own tasks.
Use OpenMLE-Evo as a harness over a model you already serve. The matched comparisons show it converting several frontier models into stronger MLE systems by Medal Average than Claude Code or Codex on the same models, and the bounded-context design is the transferable part.
Use OpenMLE-ERL's reward shaping if you are doing RL on any task family where scores are continuous, heterogeneous in range, and clustered near the current policy frontier. This is the general failure mode of RLVR outside mathematics, and adaptive bounds plus upper-tail weighting is a well-specified answer to it.
Do not adopt it under a commercial licence. The repository, both model repositories, and the datasets are CC BY-NC 4.0. Commercial use is not granted.¹² This is the single most consequential practical fact on this page and it is easy to miss behind the word "open".
Do not expect to reproduce the numbers from the repository alone. The sandbox job service that Section 3.4 describes, the prepared MLE-Bench data, the leaderboard metadata, and the model serving are all external inputs. The repository's own validation document states plainly that neither of its smoke tests "independently reproduces paper tables".¹³
Do not read "71.21% beats GPT-5.5 + Codex" as a capability ranking. That comparison is two medal outcomes out of 66 wide, against a single-run reference, on a row whose own standard deviation is 8.57 percentage points. See Failure modes.
Do not use it for anything outside tabular, image, text, time-series, or audio supervised learning. Classification and regression are 87% of the gym; segmentation, detection, and generation together are 4%.²

Architecture¶

The stack is organised around one contract: an operator call takes a task, an operator name, and a context built from zero or more parent programs, and returns a program that the sandbox executes for a scalar score.¹ Everything else is a choice about how to build the context, how to convert the score into a learning signal, and how to pick the next parent.

The environment layer is a job service, not a library. The search runtime and both training stages talk to it over HTTP: POST /api/v1/jobs with the program, a data_dir, a resource_type of cpu or gpu, a priority, and a timeout, then poll GET /api/v1/jobs/{job_id} until the status leaves queued/running for one of completed, failed, or cancelled.¹⁴ Six feedback modes come back: success, runtime error, missing code, missing submission, scoring failure, and timeout, each with logs, error class, runtime metadata, and artifacts, so the agent can distinguish a broken program from a weak one.² This is the same isolation posture described in agent sandboxing and isolation, implemented as a scheduler dispatching to CPU and GPU Docker workers.

The learning layer solves a problem that short-horizon RLVR does not have. One task optimises accuracy, another log loss; raw scores are not comparable, and leaderboard or theoretical extrema are usually far wider than the band the current policy occupies, so meaningfully different programs collapse to nearly identical rewards. OpenMLE derives bounds from the policy's own recent score frontier instead: the best observed score sets the top, the 16th-best sets the bottom, the bottom is then extended downward by a quarter of that gap so that moderately good programs are not clipped to zero, and the result is clamped inside the task's declared valid range.⁸ On top of that, an entropic advantage replaces GRPO-style group normalisation, concentrating the update on the top of each rollout group at a fixed KL budget from uniform.⁸

The search layer keeps a deterministic experience card per evaluated node (provenance, score, delta against parent, method family, rank, runtime, error signature) and aggregates them into a task-global board.⁴ Parent selection is a softmax over a three-term utility of normalised score, normalised improvement over the strongest parent, and method-family novelty, so a branch that is behind on absolute score but leading on recent gain stays selectable.⁴ Natural-language memory is synthesised only after an operator has chosen its nodes, which is what produces the large tail compression in prompt length.

The reward transform, executed¶

The two transforms that make execution feedback trainable are small enough to reimplement and check. The block below is the paper's Appendix B.5 in numpy, with the properties the method depends on asserted, including the two degenerate cases the shipped code guards.

"""OpenMLE-ERL reward shaping: adaptive bounds + entropic advantage (numpy only).
Reimplements Appendix B.5 of arXiv 2607.28568 and asserts the properties the
paper relies on, including the two edge cases the shipped code guards."""
import math
import numpy as np

TARGET_KL = math.log(2.0)          # ttt_reward_postprocess.py hard-codes log 2
LOWER_SHIFT = 0.25                 # ADAPTIVE_REWARD_BOUND_LOWER_SHIFT_RATIO
TOP_K = 16                         # ADAPTIVE_REWARD_BOUND_MODE=top1_top16


def adaptive_bounds(signed_scores, static_best, static_worst):
    """B = x_(1); W = x_(min(16,K)) extended down by 25% of the gap, then
    clamped inside the task's static (leaderboard/theoretical) bounds."""
    xs = sorted(map(float, signed_scores), reverse=True)
    best = xs[0]
    worst = xs[min(TOP_K, len(xs)) - 1]
    worst -= LOWER_SHIFT * max(best - worst, 0.0)
    best, worst = min(best, static_best), max(worst, static_worst)
    return best, (worst if worst < best else best - 1e-6)


def processed_reward(signed_score, best, worst):
    """Eq. 1 with alpha = 1: linear map into [0, 1], clipped."""
    return float(np.clip((signed_score - worst) / (best - worst), 0.0, 1.0))


def entropic_advantage(rewards, iters=60, eps=1e-12):
    """Eq. 2: softmax weights at the beta that spends exactly TARGET_KL of
    divergence from uniform, scored against a leave-one-out denominator."""
    r = np.asarray(rewards, dtype=np.float64)
    if r.size < 2 or np.allclose(r, r[0]):
        return np.zeros(r.size)
    centred = r - r.max()

    def kl(beta):
        logq = beta * centred - np.logaddexp.reduce(beta * centred)
        return float(np.sum(np.exp(logq) * (logq + math.log(r.size))))

    lo, hi = 0.0, 1.0
    while hi < 1e6 and kl(hi) < TARGET_KL:
        hi *= 2.0
    for _ in range(iters):
        mid = 0.5 * (lo + hi)
        lo, hi = (mid, hi) if kl(mid) < TARGET_KL else (lo, mid)
    weights = np.exp(hi * centred)
    loo = (weights.sum() - weights) / (r.size - 1)
    return weights / (loo + eps) - 1.0


def grpo_advantage(rewards, eps=1e-6):
    a = np.asarray(rewards, dtype=np.float64)
    a = a - a.mean()
    return a / (a.std(ddof=1) + eps)


# A log-loss task: 16 programs inside a narrow band, leaderboard range far wider.
LOG_LOSS = [0.171, 0.164, 0.159, 0.152, 0.148, 0.141, 0.139, 0.131,
            0.129, 0.122, 0.118, 0.111, 0.104, 0.098, 0.091, 0.083]
signed = [-x for x in LOG_LOSS]                    # negate: larger is better
STATIC = (0.0, -20.0)                              # perfect run vs a terrible one

static_r = np.array([processed_reward(s, *STATIC) for s in signed])
best, worst = adaptive_bounds(signed, *STATIC)
adaptive_r = np.array([processed_reward(s, best, worst) for s in signed])

# 1. Static bounds destroy the resolution the policy actually needs.
assert np.ptp(static_r) < 0.01, np.ptp(static_r)
assert np.ptp(adaptive_r) == 0.8, np.ptp(adaptive_r)
# 2. The 25% shift is what keeps the weakest member off the zero floor.
assert adaptive_r.min() > 0.0
no_shift_best, no_shift_worst = max(signed), min(signed)
assert processed_reward(min(signed), no_shift_best, no_shift_worst) == 0.0
# 3. Bounds never escape the task's declared valid range.
assert STATIC[1] <= worst < best <= STATIC[0]

ent, grp = entropic_advantage(adaptive_r), grpo_advantage(adaptive_r)
top = int(np.argmax(adaptive_r))
# 4. Entropic weighting concentrates the update on the best candidate.
assert np.argmax(ent) == top and ent[top] > grp[top]
assert np.all(np.diff(ent[np.argsort(adaptive_r)]) >= -1e-12)   # monotone
# 5. The KL budget is spent exactly, not approximately.
q = (ent + 1.0) / (len(ent) - 1 + ent + 1.0)
q = q / q.sum()
assert abs(float(np.sum(q * np.log(q * len(q)))) - TARGET_KL) < 1e-6
# 6. Degenerate groups produce no gradient rather than a divide-by-zero.
assert entropic_advantage([0.7]).tolist() == [0.0]
assert np.array_equal(entropic_advantage([0.4] * 8), np.zeros(8))

print(f"static bounds  -> reward spread {np.ptp(static_r):.4f}")
print(f"adaptive bounds [{worst:.4f}, {best:.4f}] -> reward spread {np.ptp(adaptive_r):.4f}")
print(f"best candidate: GRPO advantage {grp[top]:.3f} -> entropic {ent[top]:.3f} "
      f"({ent[top] / grp[top]:.2f}x)")

Output:

static bounds  -> reward spread 0.0044
adaptive bounds [-0.1930, -0.0830] -> reward spread 0.8000
best candidate: GRPO advantage 1.718 -> entropic 5.654 (3.29x)

The first two lines are the whole argument for adaptive bounds: with leaderboard-width bounds, sixteen genuinely different programs occupy 0.44% of the reward range and the policy cannot tell them apart; with policy-derived bounds they occupy 80%. The third line quantifies the upper-tail effect the paper reports as 4.0x in Figure 8a. Sampling 4,000 random sixteen-member groups against the shipped implementation puts that ratio between 1.67x and 6.34x with a median of 2.96x, so 4.0x is a reachable and unremarkable value rather than a tuned one.²⁹

How to use it¶

The search runtime is the entry point most people want, and it is deliberately not a single command. OpenMLE-Evo orchestrates search and scoring; it does not serve models, prepare benchmark data, or ship sandbox images.¹⁵ Before it will run you need Python 3.11 or 3.12, an OpenAI-compatible model endpoint, a sandbox implementing the job protocol, the evaluation parquet, prepared task data, and leaderboard metadata.

cd OpenMLE-Evo
python3.12 -m venv .venv && source .venv/bin/activate
python -m pip install -r requirements.txt      # installs tts_search + vendored aira-evo
cp .env.example .env                           # then fill in the endpoints below

The variables that matter are SGLANG_BASE_URL (must end in /v1), OPENMLE_MODEL_ID, SANDBOX_URL for the single-worker profile or SANDBOX_ROUTER_URL for the multi-worker one, and the two sandbox API keys.¹⁵ Verify both services before launching anything long:

curl -fsS "${SGLANG_BASE_URL}/models"
curl -fsS "${SANDBOX_ROUTER_URL}/health"
curl -fsS -H "X-API-Key: ${SANDBOX_GPU_API_KEY}" "${SANDBOX_ROUTER_URL}/api/v1/workers/status"

Then run the smoke test on one task before committing a full evaluation, because a full evaluation is 22 tasks times 3 epochs times a 12-hour sandbox budget:

OPENMLE_CONFIG_NAME=experiment/openmle_evo_smoke \
  ./scripts/run_standard.sh 'search.runner.task_list=[spooky-author-identification]'

The six documented success criteria are worth checking rather than eyeballing: the manifest shows the expected profile and worker counts, status_count.success in stat.json exceeds zero, every successful step carries status_code: 200 and non-empty token statistics, router jobs reach completed with distinct worker_id values in multi-worker mode, submit_score is populated and submission.csv passes the scorer, and the process exits zero.¹⁵

For a full run, the shipped experiment/openmle_evo profile sets max_steps=800, time_budget=43200, model_plus_sandbox_time_budget=64800, n_samples_per_task=3, and execution_timeout=7200.¹⁵ Read those two budgets carefully. The paper's headline "12-hour per-task budget" is time_budget, the sandbox execution allowance; the wall-clock ceiling including model inference is model_plus_sandbox_time_budget, which defaults to 18 hours. Multi-GPU mode is one process with N async workers, not N processes:

AIRAEVO_WORKERS=8 ./scripts/run_multi_gpu.sh

How to develop with it¶

Building your own tasks means honouring the package contract rather than the code: raw/ for original assets, data/public/ for the description, training data, test inputs and sample submission, data/private/ for hidden answers, and utils/prepare.py plus utils/metric.py for deterministic splitting and scalar scoring.² The construction pipeline validates a package by executing prepare.py and scoring the sample submission; anything that will not build or will not produce a valid scalar is dropped before the semantic quality filter ever sees it. That ordering is the reusable lesson: check executability mechanically first, then spend LLM budget on the five semantic dimensions (task validity, data sufficiency, raw-data usage, task complexity, data quality).²

The funnel is steep and worth internalising before you plan capacity. Of roughly 11,000 competitions in the Meta Kaggle catalogue, 3,972 survived leaderboard-length, MLE-Bench-overlap, and licensing screens (36%), 2,839 built into executable packages (26%), and 2,240 passed the semantic gate (20%).² Budget for four-fifths attrition.

Extending the operator set is the deeper customisation, and it is a two-sided change. An operator is simultaneously a training target in OpenMLE-ERL (with a sampling probability: Draft 0.50, Improve 0.17, Debug 0.17, Crossover 0.16) and a prompt template plus retrieval policy in OpenMLE-Evo.¹⁶ Adding one without the other reproduces exactly the split the paper is arguing against.

For RL, the released layout is unusual and the README explains why: the core modules sit at the directory root because SLIME imports custom functions by module path, so repackaging them breaks the runtime import contract.¹⁷ Install into a pinned SLIME checkout rather than the other way round:

git clone https://github.com/THUDM/slime.git && cd slime
git checkout 680824dd5e01a2e83750bf87fc366ec6fa98766c
cp -a /path/to/OpenRSI/OpenMLE-ERL/RL examples/openmle_rl
python -m pip install -r examples/openmle_rl/requirements.txt

Four launch profiles ship. The 30B model trains on a single node (8 GPUs colocated for sync, or 4 training plus 4 rollout for async); the 35B model needs two nodes (16 GPUs colocated, or 8 training plus 8 rollout).¹⁷ That is the real hardware floor for reproduction, and it is entirely separate from the RTX 4090 in the headline.

How to maintain it¶

Three things drift and each has a cheap check.

The reward variant. The launchers expose a REWARD_VARIANT switch. Only newreward uses the adaptive bounds plus entropic advantage the paper describes; medal_binary sets ENABLE_DYNAMIC_SCORE_BOUNDS=0 and USE_TTT_DISCOVER_ADVANTAGE=0, and human_rank swaps the mapping mode.¹⁶ If you inherit a run and the reward curve looks unlike Figure 6, check that variable before anything else.

The sandbox contract. Any sandbox you build must return job_id on submit and a status field that eventually leaves queued/running. The client retries 429, 500, 502, 503, and 504 with exponential backoff and honours Retry-After up to 30 seconds; it does not retry 4xx.¹⁴ A sandbox that signals overload with 400 will fail the whole search rather than backing off.

The scoring trust boundary. sandbox.trust_model_validation_score defaults to false, so a score the model prints to stdout is recorded in raw_scores but never used for selection; selection uses the sandbox's own validation score.¹⁵ Flipping this to true reintroduces the most direct reward-hacking channel there is. Treat it as a one-way door and see evaluation integrity and anti-gaming.

On that subject, the paper documents an actual reward-hacking incident and its fix: smaller models learned to shuffle the sample submission and resubmit it, plateauing the reward at a low level, so the team inserted o3-mini as a pre-execution judge that assigns -0.5 and skips the sandbox when it detects a hack.¹⁸ Two things follow. Your reward curve plateauing low is a hacking signal, not necessarily a capacity signal. And the "open" RL loop has a proprietary third-party model wired into it, which you will need to substitute.

How to run it in production¶

Separate the two compute pools, because the paper's headline conflates them in a way that is easy to misread. The sandbox pool is where candidate ML programs train and predict: one RTX 4090 capped at 12 GB of VRAM per task, 12 hours of it, with a 2-hour per-program execution timeout.¹⁹ The serving pool is where the 35B MoE generates programs, and the paper explicitly excludes it from the budget comparison: footnote 3 states the comparison "does not compare model-inference cost", and the Table 11 caption notes that "remote LLM-serving compute and CPU/RAM are omitted".¹⁰ Anyone planning capacity from the abstract alone will under-provision by the entire inference tier.

Size the sandbox pool from the funnel, not the task count. With n_samples_per_task=3 and max_steps=800, a 22-task evaluation is up to 52,800 sandbox jobs, each allowed 2 hours, drawn against a 12-hour per-task sandbox budget. The scheduler, not the search loop, is what keeps that tractable, and the search runtime is explicit that time spent waiting on a GPU pool does not count against the search budget while max_wall_time_secs remains the hard cap.¹⁵

For the training side, the async rollout is the one change with a measured payoff. Because sandbox execution dominates latency and runtimes vary by orders of magnitude across tasks, a synchronous batch idles until its slowest job returns. Launching generation-and-execution groups independently and consuming completed groups from a queue cut mean step time from 97.0 to 50.8 minutes across 40 matched steps, a 1.91x speedup, without unbalancing task exposure (per-task step counts stayed within 2 steps of the median, coefficients of variation 1.56% and 2.06%).²⁰ The main text keeps this claim qualitative; the number is in Appendix B.4. This is the same throughput argument made in RL rollout redundancy and applies to any RL loop whose verifier is slow and variable.

The released training path was smoke-validated on 8 H200 GPUs, completing checkpoint restore, two rollout-and-optimiser steps, weight synchronisation to SGLang, and evaluation. The README is careful to say this "does not claim exact paper reproduction".¹⁷ Plan a reproduction attempt as an engineering project, not a script run, and see RL cluster bring-up for the underlying platform work.

Failure modes¶

Reading the frontier comparison as a capability result. "Reaches 71.21%, exceeding GPT-5.5 + Codex" is a 3.03-point margin, which on a 22-task benchmark averaged over 3 runs is exactly 2 medal outcomes out of 66 (47 versus 45).²¹ The Frontis row's own standard deviation is 8.57 points, 2.8 times the margin, and the reference configurations were run once because of cost, so they carry no dispersion at all.²² The same caution applies to "matches Kimi K3's Gold rate". The post-training comparison is the load-bearing result; the frontier comparison is not.
Assuming the domain harness always wins. The paper states that OpenMLE-Evo "consistently converts the same model into a stronger MLE system than general-purpose coding-agent harnesses". Its own table contradicts this on GLM-5.2, where Claude Code beats OpenMLE-Evo on two of three metrics: Valid Rate 21.00 versus 19.67 and Human Rank 0.7948 versus 0.7069, with only Medal Average favouring OpenMLE-Evo at 62.12% versus 59.09%.²³ The consistency claim holds for Medal Average across the four matched models, not for the metric set. Evo-Max recovers the loss, but Evo-Max is a different system.
Treating NatureBench Lite as an independent held-out benchmark. NatureBench is a benchmark from the same organisation: 11 of its 17 authors are also authors of this paper, including both project leaders and the corresponding author, and both repositories live under the FrontisAI GitHub organisation.²⁴ Figure 2's caption nonetheless describes the work as "evaluated only on third-party benchmarks". MLE-Bench is genuinely third-party (OpenAI); NatureBench is not, and the transfer claim should be read as in-house generalisation evidence.
Over-reading the ten-task transfer numbers. On NatureBench Lite each task is worth 10 percentage points, so "Match-SOTA from 50% to 70%" is 2 tasks and "Surpass-SOTA from 20% to 30%" is 1 task.²⁵ The paper discloses this in Appendix D.2 and the repository repeats it. Note also that the harness comparison there is against original AIRA-Evo only; the base model was never run under Claude Code on NatureBench, and the published NatureBench leaderboard has Claude Opus 4.7 and GLM-5.2 with Claude Code at 100% Match-SOTA on the same ten-task subset.²⁵
Reading "5,758 tasks" as 5,758 downloadable tasks. Full task-package data is released for 1,415 tasks; for the remaining 4,343 only prepare.py and metric.py reconstruction scripts ship, because of source-data licensing.²⁶ Rebuilding those requires Kaggle access and working upstream assets, and the paper does not report how many still rebuild successfully.
Expecting the sandbox service in the repository. Section 3.4 describes a centralised scheduler dispatching to CPU and GPU Docker workers, and Table 11 gives OpenMLE a tick for Sandbox. What the repository actually contains for that protocol is clients: three copies of an httpx caller for /api/v1/jobs under OpenMLE-Evo, OpenMLE-ERL/RL, and OpenMLE-ERL/SFT. The only file named for sandbox execution, OpenMLE-Gym/openmle_gym/sandbox_exec.py, is 29 lines that importlib-load a prepare.py and call it in-process during task construction; it does not isolate anything.²⁷ You are building the job service.
Trusting the RL reward on sparse-success tasks. All four shipped launchers set ADAPTIVE_BOUND_SINGLE_FINITE_REWARD_ONE=1, which returns reward exactly 1.0 whenever a rollout group contains a single finite score, regardless of how bad that score is. Executed against the shipped code, a lone score of 0.001 on a task whose leaderboard range is 0.30 to 0.99 receives reward 1.0; with the flag off the same score maps to 0.2985.²⁸ The paper's Appendix B.5 does not mention this branch, and it inverts the stated design principle that "a barely viable submission should not receive the same positive reward as a top-performing one" precisely on the hard tasks where sparse success is the norm.
Citing the search-efficiency result as a discovery-rate result. The 84.3% improvement is new-best updates per million tokens. The absolute count of new-best validation updates rose only from 229 to 246, a 7.4% increase, while token consumption fell 41.7%.¹¹ The finding is that each expansion got cheaper, which the paper says plainly; a summary that drops the denominator overstates it.
Assuming multimodal capability carries over. The 35B checkpoint retains its base model's vision and MTP components, but the release notes state that OpenMLE post-training and every reported evaluation cover text and code behaviour only.²⁶ Image-conditioned behaviour is inherited and unmeasured.
Missing the licence. CC BY-NC 4.0 on the code, both model repositories, and the datasets. Vendored AIRA-Dojo material retains its own terms.¹²

References¶

Yang, Jiang, Fu, Luo, Ren, Wang, Zhao, Liu, Zuo, Wang, Fan, Tian, Yuan, Lin, Sheng, Qiang, Jia, Lv, Hua, Lei, Sun, Ding, Zhou, and Zhang, "Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering" (arXiv 2607.28568): https://arxiv.org/abs/2607.28568
OpenRSI repository (OpenMLE-Gym, OpenMLE-ERL, OpenMLE-Evo): https://github.com/FrontisAI/OpenRSI
OpenRSI results, training, and release-boundary documents: https://github.com/FrontisAI/OpenRSI/tree/main/docs
Frontis-MA1-35B weights: https://huggingface.co/FrontisAI/Frontis-MA1-35B
Frontis-MA1-30B weights: https://huggingface.co/FrontisAI/Frontis-MA1-30B
OpenMLE Tasks dataset: https://huggingface.co/datasets/FrontisAI/OpenMLE-Tasks
OpenMLE SFT Traces dataset: https://huggingface.co/datasets/FrontisAI/OpenMLE-SFT-Traces
Chan et al., "MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering" (arXiv 2410.07095): https://arxiv.org/abs/2410.07095
Wang et al., "NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?" (arXiv 2606.24530): https://arxiv.org/abs/2606.24530
SLIME training framework (pinned at 680824dd for reproduction): https://github.com/THUDM/slime

arXiv 2607.28568v1, Abstract and Section 1. "We introduce OpenMLE, an open full-stack system for RSI research in MLE, spanning verifiable task environments with execution feedback (OpenMLE-Gym), operator learning (OpenMLE-ERL), and long-horizon search (OpenMLE-Evo)." Section 2 formalises the contract: at step t the search selects operator a_t and context c_t, the model proposes p_t ~ g_theta(. | tau, a_t, c_t), and the sandbox returns s_t = R_tau(E(p_t, tau)). Section 4.1: "Rather than training full trajectories, OpenMLE trains a compact set of reusable program-transformation operators over executable candidates." The arXiv abstract landing page writes "OpenMLE-RL" where the PDF, the repository directory OpenMLE-ERL/, and all configs write "OpenMLE-ERL". ↩↩
arXiv 2607.28568v1, Section 3 and Figure 3/Figure 4. 5,758 executable tasks = 156 curated anchors + 3,362 Kaggle-dataset tasks + 2,240 Kaggle-competition tasks. Competition funnel: ~11,000 Meta Kaggle competitions, 3,972 eligible candidates (36% retained) after leaderboard-length, MLE-Bench-exclusion, and licensing screens, 2,839 executable packages (26%), 2,240 quality-gated (20%). Package layout: raw/, data/public/ (description.txt, train, test, sample_submission.csv), data/private/test_answer.csv, utils/prepare.py, utils/metric.py. Quality filter dimensions: task validity, data sufficiency, raw-data usage, task complexity, data quality. Modality mix: tabular 44%, image 18%, time series 13%, multimodal 11%, text 9%, audio 2%, video 1%, other 2%. Task types: classification 56%, regression 31%, other 9%, segmentation 2%, object detection 1%, generation 1%. Section 3.4 lists the six feedback modes: successful completion, runtime error, missing code, missing submission, scoring failure, timeout. ↩↩↩↩↩↩
arXiv 2607.28568v1, Section 4.2 and Appendix B.1. Parallel path contributes 17,245 full-response Draft examples; evolutionary path contributes 9,014 trajectory-step examples; total released SFT corpus 26,259 examples under a budget-adaptive stopping rule (stop at an accepted-example quota or when the task exhausts its execution budget). ↩
arXiv 2607.28568v1, Section 5. Experience card fields are extracted deterministically from search state and execution result (provenance, performance, execution outcome, resource usage); the task-global board aggregates method families, family-wise bests, underexplored directions, repeated failures, score trends, and the parent graph. Parent selection: U_i = lambda_s * s_i + lambda_delta * Delta_i + lambda_n * nu_i with softmax sampling at temperature tau (Eq. 4). Section 5.3: original AIRA-Evo "eagerly invokes a language model to summarize the history of every evaluated node by default"; OpenMLE-Evo defers synthesis until an Improve, Crossover, or Debug call has selected its nodes. ↩↩↩
Hugging Face model API, retrieved 2026-08-01. FrontisAI/Frontis-MA1-35B: 35,951,822,704 BF16 parameters, architecture Qwen3_5MoeForConditionalGeneration, model_type qwen3_5_moe, pipeline tag image-text-to-text, base model Qwen/Qwen3.6-35B-A3B, licence cc-by-nc-4.0. FrontisAI/Frontis-MA1-30B: 30,532,122,624 BF16 parameters, Qwen3MoeForCausalLM, num_experts 128 with num_experts_per_tok 8, base model Qwen/Qwen3-30B-A3B-Thinking-2507, licence cc-by-nc-4.0. The A3B suffix in both base models denotes roughly 3B active parameters per token. ↩
arXiv 2607.28568v1, Table 1 Panel A. Qwen3.6-35B-A3B + OpenMLE-Evo: 19.67/22 Valid, 39.39% Medal Average, 0.5828 Human Rank. Frontis-MA1-35B + OpenMLE-Evo: 21.67/22, 60.61%, 0.7647. Frontis-MA1-35B + OpenMLE-Evo-Max: 22.00/22, 71.21%, 0.8126. Qwen3-30B-A3B-Thinking-2507 + OpenMLE-Evo: 17.33/22, 34.85%, 0.5573. Frontis-MA1-30B + OpenMLE-Evo: 21.67/22, 53.03%, 0.7055. Frontis-MA1-30B + OpenMLE-Evo-Max: 22.00/22, 66.67%, 0.8053. Matched AIRA-Evo comparison on Frontis-MA1-35B: 53.03% versus 60.61% under OpenMLE-Evo. ↩
Derived on this page from the standard deviations in Table 9, not stated in the paper, which reports no significance tests. With three evaluation epochs per configuration, the standard error of the difference is sqrt(s1^2/3 + s2^2/3). 35B pair: 60.61% +/- 7.73% versus 39.39% +/- 5.67% gives a 21.22-point gap against a 5.53-point standard error, about 3.83 standard errors. 30B pair: 53.03% +/- 4.29% versus 34.85% +/- 2.14% gives 18.18 points against 2.77, about 6.57. By the same arithmetic the Frontis-MA1-35B versus GLM-5.2 comparison under OpenMLE-Evo-Max (71.21% +/- 8.57% versus 66.67% +/- 8.57%) is 0.65 standard errors and is not distinguishable from noise. ↩
arXiv 2607.28568v1, Section 4.3 and Appendix B.5. Eq. 1: r_base(s; b_best, b_worst) = clip(((s - b_worst)/(b_best - b_worst))^alpha, 0, 1). Adaptive bounds: "The best observed score sets the upper end of the range. The 16th-best score sets the lower reference point; when fewer than 16 scores are available, we use the lowest available score", then W_dyn <- W_dyn - 0.25 * max(B_dyn - W_dyn, 0), then B = min(B_dyn, B_static), W = max(W_dyn, W_static). Eq. 2 gives the entropic advantage; the appendix specifies max-centering, beta chosen by binary search so that KL(q_beta || Unif(K)) is approximately log 2, "with maximum search value 10^6 and 60 bisection iterations", and a leave-one-out denominator. Zero advantages are returned if K < 2 or all rewards are equal. ↩↩
Verified against FrontisAI/OpenRSI at commit c16ebd6 on 2026-08-01. OpenMLE-ERL/RL/ttt_reward_postprocess.py calls compute_ttt_entropic_advantages(group_rewards, target_kl=math.log(2.0), beta_max=1e6, iters=60, eps=1e-12). All four launch configs (configs/{sync,async}_{single,multi}_node.env.example) set ENABLE_DYNAMIC_SCORE_BOUNDS=1, ADAPTIVE_REWARD_BOUND_MODE="top1_top16", and ADAPTIVE_REWARD_BOUND_LOWER_SHIFT_RATIO=0.25, matching Appendix B.5 exactly. An independent reimplementation written from the paper text was cross-checked against the shipped compute_ttt_entropic_advantages over 500 randomly generated reward groups of size 2 to 32 across three magnitude scales; maximum absolute deviation was below 1e-9. The adaptive bound pair was cross-checked against the shipped compute_adaptive_bound_pair on a signed log-loss group and agreed to 1e-9. ↩
arXiv 2607.28568v1, Appendix E and Table 11, "rechecked against the cited papers, official repositories, and the official MLE-Bench leaderboard submissions as of July 2026". Fourteen comparable systems are scored across Data, Sandbox, Train code, RL method, Eval, and Weights; OpenMLE is the only row with all six ticks. The same table lists higher MLE-Bench Lite Medal Rates for other systems under different budgets and accelerators: Famou-Agent 2.0 80.30% (24h, 1xA800, Gemini-3-Pro-Preview), MLEvolve 80.30% (12h, 1xH200, Gemini-3-Pro-Preview), AIBuildAI 77.27% (24h, 1xA100, Claude-Opus-4.6), ML-Master 2.0 75.76% (24h, 2xRTX 4090, DeepSeek-V3.2-Speciale), against OpenMLE 71.21% (12h, RTX 4090 at 12 GB VRAM). The caption states that "Scores are not strictly comparable because the systems differ in backbone model, compute and wall-clock budgets, hardware, external-resource access, number of runs, and aggregation procedure" and that "Run settings report the per-task wall-clock limit and sandbox GPU allocation; remote LLM-serving compute and CPU/RAM are omitted". Section 6.1 footnote 3: the budget comparison "uses the per-run evaluation compute configurations ... reported in the official MLE-Bench runs registry. It does not compare model-inference cost or normalize different accelerators to FLOPs." ↩↩
arXiv 2607.28568v1, Section 6.5 and Figure 16, over 66 matched task-runs with the same Frontis-MA1-35B checkpoint, seed, and 12-hour budget. Total model tokens 129.3M -> 75.3M (-41.7%); prompt tokens 83.5M -> 41.5M (-50.3%); evaluated nodes 3,430 -> 3,004 (-12.4%); new-best validation updates 229 -> 246 (+7.4%); new-best updates per 1M model tokens 1.77 -> 3.27 (+84.3%); Improve operations setting a new best 44/931 (4.73%) -> 72/769 (9.36%). Improve prompt mean 102.8K -> 35.7K characters (-65.3%), 99th percentile 389.0K -> 54.3K (-86.1%); Crossover prompt mean 140.4K -> 55.3K (-60.6%), 99th percentile 419.2K -> 78.4K (-81.3%). ↩↩
FrontisAI/OpenRSI README.md and LICENSE: "Original OpenRSI and OpenMLE material in this repository is released under CC BY-NC 4.0 for attribution-required, non-commercial use. Commercial use is not granted by this license." Both Hugging Face model repositories carry the license:cc-by-nc-4.0 tag. OpenMLE-Evo/README.md adds that "Vendored AIRA-Dojo material retains its own license". ↩↩
FrontisAI/OpenRSI OpenMLE-Evo/docs/validation.md, "External validation boundary": "The MLE-Bench smoke establishes that the entrypoint, generation, concurrency, router, sandbox, scoring, checkpoint, output, and final-submit paths work together. NatureBench static/unit validation establishes configuration and adapter behavior; a full NatureBench score additionally requires its external task packages, eval service, SCM host, and container image. Neither validation independently reproduces paper tables." GPU smoke validation was performed 2026-07-28. ↩
FrontisAI/OpenRSI OpenMLE-Evo/tts_search/reward_func_utils.py at commit c16ebd6. Submit payload fields: name, code, data_dir, timeout, resource_type (cpu or gpu), priority (1 high, 2 low), and an environment map; auth via the X-API-Key header. Polling waits while status is in ["running", "queued"] and returns on ["completed", "failed", "cancelled"]. Retryable submit statuses are {429, 500, 502, 503, 504} with up to 50 attempts, exponential backoff, and Retry-After honoured up to 30 seconds; poll retries cover {502, 503, 504, 429} up to 20 times; 4xx on poll returns immediately. Config defaults in tts_search/configs/sandbox/default.yaml: concurrency 66, job_timeout 7200, wait_timeout 86400, poll_interval 10. ↩↩
FrontisAI/OpenRSI OpenMLE-Evo/docs/usage.md. Prerequisites: Python 3.11 or 3.12, an OpenAI-compatible model service, "a GPU/CPU sandbox compatible with the /api/v1/jobs protocol", the evaluation parquet, prepared task data, leaderboard metadata, and sandbox API keys. Section 8 default full-evaluation profile experiment/openmle_evo: max_steps=800, time_budget=43200, model_plus_sandbox_time_budget=64800, n_samples_per_task=3, evaluation_protocol=self_valid, execution_timeout=7200. Section 7 lists the six smoke success criteria. Section 6: "Time spent waiting on GPU/resource pools does not count against the effective search budget; max_wall_time_secs remains the hard cap on total wall-clock time." Section 3 secure defaults: sandbox.verify_tls=true and sandbox.trust_model_validation_score=false, with "self-valid stdout scores are only recorded in raw_scores, and search selection uses the validation/score returned by the sandbox". Section 12: multi-GPU mode is a single-process async multi-worker by design. ↩↩↩↩↩↩
arXiv 2607.28568v1, Table 5, and FrontisAI/OpenRSI OpenMLE-ERL/RL/configs/*.env.example plus scripts/run_openmle_rl_*.sh. Operator sampling probabilities Draft 0.50, Improve 0.17, Debug 0.17, Crossover 0.16. Rollout shape 16 prompts x 16 samples, global batch size 128, 2 optimizer steps per rollout, temperature 1.0, maximum response length 24,576 tokens. Objective GSPO with TTT-Discover-style reward post-processing, clip epsilon 3.5e-4, TIS enabled; Adam at 1e-6 constant, weight decay 0.1, beta1 0.9, beta2 0.98. The REWARD_VARIANT switch selects newreward (adaptive bounds plus entropic advantage), medal_binary (which exports ENABLE_DYNAMIC_SCORE_BOUNDS="0" and USE_TTT_DISCOVER_ADVANTAGE="0"), or human_rank. ↩↩
FrontisAI/OpenRSI OpenMLE-ERL/RL/docs/usage.md. "The single-node asynchronous profile completed real checkpoint restore, two OpenMLE-ERL/Evo rollout-and-optimizer steps, post-step weight synchronization to SGLang, and evaluation on 8 H200 GPUs. This demonstrates that the released training path runs end to end; it does not claim exact paper reproduction." Pinned runtime: THUDM/slime at commit 680824dd5e01a2e83750bf87fc366ec6fa98766c, image slimerl/slime:qwen35-qwen36-nightly-dev-20260629a-20260701. Launch topologies: single-node sync 1x8 GPUs colocated (Qwen3-30B-A3B-Thinking-2507), single-node async 4 training + 4 rollout, multi-node sync 2x8 GPUs colocated (Qwen3.6-35B-A3B), multi-node async 8 training + 8 rollout. "The core Python modules intentionally remain at the directory root because SLIME imports custom functions by module path." SFT settings from Table 4: full-parameter SFT on SLIME with Ray and Megatron-LM, 32,768-token context cutoff, bfloat16, global batch 128, learning rate 3.0e-5, cosine decay with 0.1 warmup, 3 epochs. ↩↩↩
arXiv 2607.28568v1, Appendix B.6. "We observed that our models, particularly the smaller ones used in our early experiments, experience a specific issue during RL on difficult tasks. The reward quickly plateaus at a very low level ... the models are exhibiting significant reward hacking behavior. A representative example of this is when a model takes the sample submission, randomly shuffles it, and submits it as a solution." Mitigation: "We use o3-mini as an LLM judge during the RL process. Before the code is executed in the sandbox, the judge performs a reward hack check. If reward hacking is detected, the code bypasses sandbox execution and is assigned a reward of -0.5." ↩
arXiv 2607.28568v1, Section 6.1. "Each OpenMLE-Evo configuration is evaluated with three independent runs under a fixed per-task budget of 12 hours on a single RTX 4090 (12 GB VRAM), a smaller per-task sandbox-compute budget than that used by the vast majority of reported evaluations on MLE-Bench." Evaluation set is the official 22-task MLE-Bench Lite split. Metrics: Valid Rate (mean tasks out of 22 producing a valid submission), Medal Average (mean fraction of tasks receiving any Kaggle medal), Human Rank (mean fraction of human leaderboard participants surpassed). OpenMLE-Evo-Max adds MLE-Bench-disjoint cross-task experience priors distilled from public competition artifacts and asynchronous multi-GPU parallel search, with total sandbox compute held constant. ↩
arXiv 2607.28568v1, Appendix B.4 and Figure 23. "Across the 40 matched steps, mean step time is 97.0 minutes for the synchronous run and 50.8 minutes for the asynchronous run, corresponding to a 1.91x ratio ... in two representative asynchronous runs, per-task step counts stay within +/-2 steps of the run median, with coefficients of variation of 1.56% and 2.06%." Section 4.3 keeps the main-text claim qualitative: "Because the realized speedup depends on the task-runtime distribution and worker allocation, we keep the main-text claim qualitative and report the measured timing study in Appendix B.4." ↩
Derived on this page from Table 1 and the evaluation protocol. Medal Average is a fraction of 22 tasks averaged over 3 runs, so the resolution is 1/66 = 1.5152 percentage points. 71.21% = 47/66 and 68.18% = 15/22 = 45/66, a difference of exactly 2 medal outcomes. The same arithmetic reproduces the paper's own count elsewhere: 60.61% - 39.39% = 21.22 points = 40/66 - 26/66 = 14 outcomes, matching Section 6.6's "The 14 additional medals are distributed across every group". ↩
arXiv 2607.28568v1, Appendix D.1 and Table 9. "Owing to the substantial inference and sandbox cost, Codex, Claude Code, and Gemini CLI references were evaluated only once and are therefore retained as point estimates in the main results table ... All reported +/- values denote standard deviations rather than confidence intervals." Frontis-MA1-35B + OpenMLE-Evo-Max: 71.21% +/- 8.57%. Frontis-MA1-35B + OpenMLE-Evo: 60.61% +/- 7.73%. Qwen3.6-35B-A3B + OpenMLE-Evo: 39.39% +/- 5.67%. GLM-5.2 + OpenMLE-Evo-Max: 66.67% +/- 8.57%. MiniMax M3 + OpenMLE-Evo: 59.09% +/- 0.00%. ↩
arXiv 2607.28568v1, Table 1, matched harness comparison for GLM-5.2. Claude Code: 21.00/22 Valid Rate, 59.09% Medal Average, 0.7948 Human Rank. OpenMLE-Evo: 19.67/22, 62.12%, 0.7069. OpenMLE-Evo-Max: 22.00/22, 66.67%, 0.8164. Section 6.2 states: "Across multiple model families, OpenMLE-Evo consistently converts the same model into a stronger MLE system than general-purpose coding-agent harnesses such as Claude Code and Codex." The other three matched pairs (MiniMax M3 versus Codex, Kimi K2.6 versus Claude Code, MiniMax M2.7 versus Claude Code) do favour OpenMLE-Evo on all three metrics. ↩
Author lists compared 2026-08-01. NatureBench (arXiv 2606.24530) lists Yuru Wang, Lejun Cheng, Yuxin Zuo, Sihang Zeng, Bingxiang He, Che Jiang, Junlin Yang, Yuchong Wang, Kaikai Zhao, Weifeng Huang, Kai Tian, Zhenzhao Yuan, Jincheng Zhong, Weizhi Wang, Ning Ding, Bowen Zhou, Kaiyan Zhang (17 authors). Eleven of them also author arXiv 2607.28568: Yuru Wang, Yuxin Zuo, Che Jiang, Junlin Yang, Kaikai Zhao, Kai Tian, Zhenzhao Yuan, Weizhi Wang, Ning Ding, Bowen Zhou, Kaiyan Zhang. Junlin Yang and Che Jiang are the Frontis-MA1 project leaders and Kaiyan Zhang is its corresponding author. Both projects are hosted under github.com/FrontisAI. The OpenRSI README describes NatureBench as its own release, "later used as the held-out transfer benchmark for Frontis-MA1". Figure 2's caption in arXiv 2607.28568v1 nonetheless reads "evaluated only on third-party benchmarks". ↩
arXiv 2607.28568v1, Section 6.6, Table 2, and Appendix D.2. NatureBench Lite is a fixed 10-task subset retaining the NatureBench containers, hidden evaluator, validity rules, web-search-disabled setting, and 4-hour per-task budget. Frontis-MA1-35B + OpenMLE-Evo NB adapter: 30.0% (3/10) Surpass-SOTA, 70.0% (7/10) Match-SOTA. Qwen3.6-35B-A3B + the same adapter: 20.0% (2/10), 50.0% (5/10). Qwen3.6-35B-A3B + original AIRA-Evo: 10.0% (1/10), 20.0% (2/10). Appendix D.2: "its ten-task size means that each task changes All S or All M by ten percentage points." Table 2's reference rows put Claude Opus 4.7 + Claude Code and GLM-5.2 + Claude Code at 70.0% (7/10) Surpass and 100.0% (10/10) Match on the same subset. No Claude Code or Codex run of Qwen3.6-35B-A3B on NatureBench is reported, so the harness comparison there is against original AIRA-Evo only. ↩↩
arXiv 2607.28568v1, Section 4 footnote 2, and FrontisAI/OpenRSI docs/release.md. "Owing to source-data licensing and copyright constraints, we release full task-package data for 1,415 tasks. For the remaining 4,343 tasks, we release the corresponding prepare.py and metric.py scripts without redistributing the source data." docs/release.md public source scope: "The reported MLE-Bench and NatureBench numbers require external task environments, sandbox services, datasets, and evaluation assets" and "The 35B repository retains upstream vision and MTP components, but OpenMLE post-training and reported evaluations cover text/code behavior." ↩↩
Verified against FrontisAI/OpenRSI at commit c16ebd6 on 2026-08-01 by cloning the repository and grepping the full tree (790 tracked files). /api/v1/jobs appears in exactly four files: the client implementations in OpenMLE-Evo/tts_search/reward_func_utils.py, OpenMLE-ERL/RL/reward_func_utils.py, and OpenMLE-ERL/SFT/tts_search/reward_func_utils.py, plus the prerequisite line in OpenMLE-Evo/docs/usage.md. No module in the repository serves that route. OpenMLE-Gym/openmle_gym/sandbox_exec.py is 29 lines: it loads a prepare.py with importlib.util.spec_from_file_location and calls its prepare() function in the current process. Section 3.4 of the paper describes a different component: "A centralized scheduler receives API requests, records each job, tracks worker availability, and dispatches requests to CPU/GPU Docker workers according to resource requirements." ↩
Verified by executing OpenMLE-ERL/RL/adaptive_reward_advantage_utils.score_to_group_adaptive_reward from FrontisAI/OpenRSI at commit c16ebd6 on 2026-08-01. All four shipped launchers set ADAPTIVE_BOUND_SINGLE_FINITE_REWARD_ONE=1. The function's branch reads: if single_finite_reward_one and len(available_signed_scores) == 1: ... return 1.0, context. With metadata {higher_is_better: True, theoretical_min: 0.0, theoretical_max: 1.0, leaderboard_min: 0.30, leaderboard_max: 0.99}, lone finite scores of 0.001, 0.05, and 0.30 each returned reward 1.0 with single_finite_reward_one_applied true; the same 0.001 score with the flag disabled returned 0.298534604637997. Appendix B.5 specifies "bound construction and edge-case handling" but does not describe this branch. Section 4.3 states the opposing principle: "MLE evaluation rewards the quality of the best program found; a barely viable submission should therefore not receive the same positive reward as a top-performing one." ↩
Derived on this page. arXiv 2607.28568v1 Figure 8a reports mean processed advantage for the best candidate in a rollout group rising from 1.58 without entropic weighting to 6.39 with it, labelled "4.0x stronger upper-tail signal". Sampling 4,000 random 16-member reward groups and comparing the shipped compute_ttt_entropic_advantages against the shipped compute_gspo_group_advantages gave best-candidate ratios ranging from 1.67x to 6.34x with a median of 2.96x, so the reported 4.04x sits inside the reachable band. The exact value depends on the reward distribution within the group and is not a tunable constant. ↩