Skip to content
Markdown

NatureBench: coding agents vs Nature-family SOTA

Scope: NatureBench (arXiv 2606.24530), a cross-discipline benchmark of 90 research tasks distilled from peer-reviewed Nature-family publications and scored against each source paper's reported state of the art, built on NatureGym, an automated pipeline that packages each paper into a per-task container to fix the environment-fragmentation problem of prior agent-on-research benchmarks. This page is the benchmark deep dive: what a task and its container are, how the SOTA-normalized improvement metric g and the g>0.1 surpass criterion are defined, what the ten-to-twelve frontier configurations actually scored, and the validated metric model behind the numbers. It is a science-domain instance of general agent evaluation, a sibling of the file-dependency Workspace-Bench, the ops-domain AIOpsLab, and the security-domain cybersecurity agent evaluation, and unlike LLM-judge benchmarks it scores through a hidden programmatic evaluator rather than a model grader, sidestepping the judge-reliability concerns entirely.

Version and roster note: the paper was submitted 2026-06-23 (v1) and revised 2026-07-06 (v2); the headline figures (90 tasks, 17.8% surpass, Claude Opus 4.7 strongest) are identical across both versions, so this page cites v2 and treats v1 as equivalent. One internal count is worth knowing: the abstract still advertises "ten frontier agent configurations," a figure carried over unedited from v1, while v2's Introduction was updated to twelve and the main results table (Table 4) lists twelve model rows across three harnesses. The paper never explicitly reconciles the two anywhere in its text. The evidence points to the ten-agent behavioral analysis (Section 5) being the original v1 roster: its percentages are byte-identical between v1 and v2, meaning it was never rerun after v2 added two new models (GLM-5.2 and MiniMax-M3, per the revision's own arXiv changelog note); the ten-agent subset therefore includes GLM-5.1 and MiniMax-M2.7 and excludes GLM-5.2 and MiniMax-M3, the reverse of a "drop the two weakest" reading. Quote "twelve" for the results table and treat the mechanism-analysis "ten" as the pre-revision roster, not a curated subset of the current one. Model and harness names are 2026-era and will date. The Python metric model below is executed and asserted (stdlib only).

flowchart TB
  subgraph GYM["NatureGym pipeline (per source paper)"]
    PAPER["Nature-family paper<br/>2022-2025, 6 journals"] --> PKG["Task package<br/>README + data/ (agent-visible)<br/>+ hidden evaluator.py + ground_truth/<br/>+ per-task Dockerfile"]
    PKG --> FW["Information firewall<br/>source method removed"]
  end
  FW --> TASK["90 tasks, 333 instances<br/>6 scientific domains"]
  TASK --> RUN["Agent under test<br/>12 model-harness configs<br/>web search disabled, 4h wall-clock"]
  RUN -->|"writes workspace/, submits prediction"| EVAL["Isolated container evaluator<br/>hidden ground truth, GPU matched"]
  EVAL --> G["g = dir * (m - m_sota) / |m_sota|<br/>surpass g>0.1 | match g>=0"]
  G --> RESULT["Best 17.8% surpass, 47.8% match<br/>(Claude Opus 4.7) | reproduce up to 30/90"]

What it is

NatureBench asks whether a coding agent can do more than reproduce a published result: given a real scientific machine-learning problem lifted from a Nature-family paper, can it match or beat that paper's reported state of the art without being told the paper's method. The benchmark is 90 tasks distilled from peer-reviewed publications across six Nature-family journals, spanning six scientific domains (cellular omics, protein biology, biomedical modeling, physical modeling, molecular design, and relational reasoning), with source papers drawn from 2022 to 2025.1 The tasks concentrate in three journals, led by Nature Machine Intelligence (36 tasks), Nature Methods (26), and Nature Computational Science (16); the remaining 12 tasks come from three other Nature-family journals.1

The construction rests on NatureGym, an automated pipeline that converts a source paper into a standardized, per-task container. Each task package (Table 1 of the paper) exposes an agent-visible problem/ tree (a README.md task definition, a data_description.md, and an inputs-only data/ directory) and hides evaluation/evaluator.py and evaluation/ground_truth/ from the agent, alongside an environment/Dockerfile and metadata.json.2 An information firewall removes the source method from every package, so an agent must discover a solution rather than transcribe the paper's.2 Scoring is programmatic: the hidden evaluator compares the agent's submitted prediction against held-out ground truth and computes a single improvement number per instance, with no model-in-the-loop grader.

Why use it

  • It measures discovery, not reproduction. The scoring is anchored to the source paper's published SOTA, so the question is not "did the agent produce a plausible artifact" but "did it match or beat a peer-reviewed result." The headline finding is that clear improvements are rare: the strongest configuration surpasses SOTA (g>0.1) on only 17.8% of tasks and matches it (g>=0) on 47.8%.4
  • It fixes environment fragmentation. The paper's stated motivation is that prior agent-on-research benchmarks "suffer from environment fragmentation that makes independent re-running fragile." NatureGym layers task-specific dependencies over a shared base image with pre-installed scientific and ML libraries, so every task is a self-contained, reproducible container rather than a bespoke setup.2 This is the same reproducibility discipline this KB expects of any evaluation harness.
  • It separates how agents succeed from whether they succeed. A method-pathway analysis labels each successful run by mechanism and finds that success is driven by methodological translation (converting a scientific task into a familiar supervised-prediction problem), which accounts for 45.5% of validated successes, rather than by scientific invention.5
  • It is honest about compute as a variable. Every task gets the same 4-hour wall-clock budget and a GPU matched to its requirement (3 tasks CPU-only, 70 on a single RTX 3090 or 4090, 17 on a single A800), so a weak result cannot be excused as an unfunded run, and "insufficient compute budget" shows up as a measured failure mode rather than an unknown.2

When to use it (and when not)

  • Use it to test whether an agent can advance beyond reproducing a known result on real scientific problems, a stricter bar than the artifact-completion gates of general agent evaluation or the file-reasoning gate of Workspace-Bench.
  • Use it when you want programmatic, judge-free scoring: the metric is a numeric comparison against hidden ground truth, so it avoids the grader-subjectivity failure modes of LLM-as-judge setups covered under judge reliability, at the cost of only working where a paper's primary metric is machine-checkable.
  • Use the container packaging as a template for your own reproducible agent tasks even if you do not adopt the task set; the shared-base-image plus per-task-Dockerfile pattern is the reusable part.
  • Do not read the 17.8% as a fixed ceiling. It is one snapshot of 2026-era models under a web-search-disabled protocol; the ranking is informative, the absolute number will move as models improve.
  • Do not assume the tasks generalize to bench science or non-computational fields. Every task is a computational problem with a machine-checkable primary metric distilled from a computational-science paper; wet-lab, theory, or qualitative results are out of scope and need a different instrument.
  • Do not conflate matching with discovering. A g in the match band (0 to 0.1) means the agent recovered the known frontier, usually by translating the problem into a supervised task; only g>0.1 is evidence of improvement, and the two are structurally different outcomes in the metric.

Architecture

Every task is a container built by NatureGym from one source paper and run under a strict protocol.2 The agent sees a natural-language task brief, a data description, and an inputs-only data directory; it does not see the evaluator, the ground truth, or the source paper's method. All agents run with web search disabled, which prevents retrieving the source dataset or paper content as a shortcut, so the benchmark measures solution discovery rather than retrieval.2 Each run is confined to an isolated NatureBench container with read/write access to a workspace/, a 4-hour wall-clock budget, and a GPU tier matched to the task's metadata.json (CPU-only, RTX 3090/4090, or A800).2

The benchmark comprises 90 tasks and 333 evaluation instances, a mean of 3.7 instances per task (median 3); a task's score averages its instance-level g values, so a single lucky instance cannot carry a task.2 When a run produces no valid submission, that instance is assigned a failure floor of g = -1.0 rather than being dropped, so non-submission is penalized, not ignored.3 This held-out-ground-truth plus information-firewall design is exactly the anti-shortcut posture of evaluation integrity: the agent cannot see the answer key, cannot fetch the paper, and cannot game the metric without actually solving the task.

How to use it

One metric carries the results: the SOTA-normalized improvement g, defined per instance i (Equation 1 of the paper) as

g_i = dir_i * (m_i - m_i_sota) / |m_i_sota|

where m_i is the agent's value on the task's primary metric, m_i_sota is the paper-reported SOTA, and dir_i is +1 when higher is better and -1 when lower is better.3 The value is a direction-aware relative gain over the published result. Three decision bands follow directly:3

  • Match-SOTA: g >= 0, the agent meets or exceeds the published result.
  • Surpass-SOTA: g > 0.1, a relative improvement of more than 10% over SOTA, the paper's operational definition of moving "beyond reproduction toward discovery."
  • Reproduce-mode success: g >= -0.05, used in a separate calibration where the agent is allowed the paper's method and must merely re-derive its result.

The evaluated set is twelve model-and-harness configurations (Table 4), scored on the "All" columns as the fraction of the 90 tasks reaching each band. All reported rates are integer task counts over 90 (17.8% is 16/90, 47.8% is 43/90):4

Configuration Surpass-SOTA (g>0.1) Match-SOTA (g>=0)
Claude Opus 4.7 (Claude Code) 17.8 47.8
GLM-5.2 (Claude Code) 15.6 41.1
Gemini 3.5 Flash (Gemini CLI) 15.6 37.8
GPT-5.5 (Codex CLI) 14.4 44.4
Claude Opus 4.6 (Claude Code) 12.2 36.7
MiniMax-M3 (Claude Code) 11.1 33.3
Qwen 3.7 Max (Claude Code) 10.0 28.9
Kimi K2.6 (Claude Code) 8.9 30.0
GPT-5.4 (Codex CLI) 8.9 27.8
GLM-5.1 (Claude Code) 7.8 28.9
DeepSeek-V4-Pro (Claude Code) 4.4 26.7
MiniMax-M2.7 (Claude Code) 1.1 13.3

The strongest agent, Claude Opus 4.7, surpasses SOTA on 17.8% of tasks and matches it on 47.8%; the weakest, MiniMax-M2.7, surpasses on 1.1% (a single task) and matches on 13.3%.4 The paper reports no aggregate mean across configurations; the per-configuration spread is 1.1% to 17.8% surpass. Per-domain, Claude Opus 4.7's surpass rate is highest on Physical modeling (30.8%) and Cellular omics (22.6%) and lowest on Relational reasoning (0.0%) and Biomedical modeling (7.1%), though per-domain denominators are small and the paper publishes domain shares graphically rather than as absolute task counts, so treat the domain splits as directional.4 As a reproduction calibration (agents given the method), Claude Opus 4.6 reproduces 30 of 90 tasks successfully (g >= -0.05) and DeepSeek-V4-Pro reproduces 21, well above their discovery rates and confirming that surpassing SOTA is much harder than re-deriving it.4

How to develop with it

The metric is small enough to validate directly, and doing so makes the paper's central distinction (matching a frontier is not discovering past it) concrete. This executed model implements g, its three decision bands, the failure floor, and the degenerate SOTA==0 case the paper leaves unhandled, then reproduces the exact 17.8% surpass and 47.8% match rates as integer task counts over 90:

# naturebench_metric.py -- validated: NatureBench's SOTA-normalized improvement
# metric g and the three decision bands it induces (surpass / match / below),
# with the paper's 17.8% surpass rate and 47.8% match rate reproduced exactly as
# integer task counts over 90 tasks. Pure stdlib (fractions).
#
# g_i = dir_i * (m_i - m_sota_i) / |m_sota_i|   (Equation 1)
#   dir_i = +1 when higher is better, -1 when lower is better.
#   g_i >= 0    : the agent MATCHES or surpasses the published SOTA.
#   g_i > 0.1   : the agent SURPASSES SOTA ("discovery"; a >10% relative gain).
#   g_i in [0, 0.1] : matches but does not surpass ("methodological translation"
#                     territory -- the paper's dominant success mechanism).
#   g_i >= -0.05: counts as a successful REPRODUCTION (reproduce-mode).
#   no valid submission -> g_fail = -1.0 (a floor, well below match).
# The task-level score averages g_i across a task's evaluation instances.

from fractions import Fraction as F

SURPASS = F(1, 10)      # g > 0.1  : surpass SOTA (strict)
MATCH = F(0)            # g >= 0   : match SOTA
REPRODUCE = F(-1, 20)   # g >= -0.05: successful reproduction
FAIL = F(-1)            # g_fail   : no valid submission


def g(m: F, m_sota: F, direction: int) -> F:
    """Equation 1: SOTA-normalized, direction-aware relative improvement.

    Undefined when the SOTA value is 0 (the normalizer |m_sota| vanishes); the
    paper gives no special handling, so we refuse the computation rather than
    silently dividing by zero -- an honest edge the benchmark leaves open.
    """
    assert direction in (+1, -1)
    if m_sota == 0:
        raise ZeroDivisionError("g undefined when SOTA==0 (|m_sota|=0); paper unhandled")
    return direction * (m - m_sota) / abs(m_sota)


def is_surpass(gi: F) -> bool:
    return gi > SURPASS


def is_match(gi: F) -> bool:
    return gi >= MATCH


def is_reproduced(gi: F) -> bool:
    return gi >= REPRODUCE


# ---------------- (a) exact-match sits at MATCH, not SURPASS ----------------
# An agent that reproduces SOTA exactly scores g=0: it MATCHES but does not surpass.
g_exact = g(F(90), F(90), +1)
assert g_exact == 0
assert is_match(g_exact) and not is_surpass(g_exact)

# The surpass test is STRICT: an agent sitting exactly on g=0.1 does NOT surpass.
assert not is_surpass(SURPASS) and is_match(SURPASS)     # g==0.1 boundary: match only

# ---------------- (b) below SOTA yields g<=0; no-submission floors at -1 ----------------
# Higher-is-better metric (dir=+1), agent scores below the SOTA value -> g<0.
g_below = g(F(80), F(100), +1)                            # (80-100)/100 = -0.2
assert g_below == F(-1, 5) and g_below < 0 and not is_match(g_below)
# A run with no valid submission is assigned the fail floor, far below match.
assert FAIL < g_below < 0
assert not is_reproduced(FAIL) and is_reproduced(g(F(96), F(100), +1))  # g=-0.04 >= -0.05

# Direction matters: for a lower-is-better metric (error/RMSE, dir=-1), beating
# SOTA means a SMALLER m, which must still yield a POSITIVE g (a real gain).
g_lower = g(F(80), F(100), -1)                            # -1*(80-100)/100 = +0.2
assert g_lower == F(1, 5) and is_surpass(g_lower)

# ---------------- (d) degenerate: SOTA==0 makes g undefined ----------------
raised = False
try:
    g(F(3), F(0), +1)
except ZeroDivisionError:
    raised = True
assert raised, "SOTA==0 must be flagged undefined, not silently divided"

# ---------------- (e) methodological translation vs discovery ----------------
# A g in the [0, 0.1] band is a MATCH (translate a known frontier into a solved
# supervised problem) but NOT a discovery; only g>0.1 is a genuine improvement.
g_translate = g(F(105), F(100), +1)                       # +0.05: matches, no surpass
g_discover = g(F(120), F(100), +1)                        # +0.20: surpasses
assert is_match(g_translate) and not is_surpass(g_translate)
assert is_match(g_discover) and is_surpass(g_discover)

# ---------------- (c) reproduce the 17.8% surpass / 47.8% match shape ----------------
# Build a 90-task set of task-level g scores: 16 surpass (>0.1), 27 more match
# (in [0,0.1]) for 43 matches total, and 47 below SOTA. The reported rates are
# exactly these integer task counts over 90.
scores = ([F(2, 10)] * 16          # 16 tasks surpass  (g=0.2 > 0.1)
          + [F(5, 100)] * 27       # 27 tasks match but do not surpass (g=0.05)
          + [F(-3, 10)] * 47)      # 47 tasks below SOTA (g=-0.3)
assert len(scores) == 90

surpass_rate = F(sum(is_surpass(s) for s in scores), len(scores))
match_rate = F(sum(is_match(s) for s in scores), len(scores))
assert surpass_rate == F(16, 90)   # == 17.777...%
assert match_rate == F(43, 90)     # == 47.777...%
assert round(float(surpass_rate) * 100, 1) == 17.8   # paper's headline
assert round(float(match_rate) * 100, 1) == 47.8     # paper's "matches on 47.8%"
# Every discovery is also a match; matches strictly exceed surpasses.
assert surpass_rate < match_rate
assert all(is_match(s) for s in scores if is_surpass(s))

print(f"OK metric g: exact-match g={g_exact} is MATCH not SURPASS; "
      f"g=0.1 boundary surpass={is_surpass(SURPASS)} (strict); "
      f"below-SOTA g={g_below} match={is_match(g_below)}; "
      f"lower-is-better beat g={g_lower} surpass={is_surpass(g_lower)}; "
      f"SOTA==0 undefined-guarded={raised}. "
      f"OK bands: translate g={g_translate} (match, not discovery) vs discover "
      f"g={g_discover} (surpass). "
      f"OK shape: surpass={surpass_rate} ({float(surpass_rate)*100:.1f}%) < "
      f"match={match_rate} ({float(match_rate)*100:.1f}%) over 90 tasks.")

Run output:

OK metric g: exact-match g=0 is MATCH not SURPASS; g=0.1 boundary surpass=False (strict); below-SOTA g=-1/5 match=False; lower-is-better beat g=1/5 surpass=True; SOTA==0 undefined-guarded=True. OK bands: translate g=1/20 (match, not discovery) vs discover g=1/5 (surpass). OK shape: surpass=8/45 (17.8%) < match=43/90 (47.8%) over 90 tasks.

The model isolates the benchmark's thesis. The surpass test is a strict g > 0.1, so an agent that lands exactly on the threshold, or exactly reproduces SOTA at g = 0, matches without discovering; the direction term makes a lower-is-better metric behave symmetrically (beating a smaller-is-better SOTA yields a positive g); and the one place the published definition is silent, a SOTA value of 0 that makes |m_sota| vanish, is a real undefined case a re-implementation must handle rather than divide through. The 16/90 and 43/90 reconstruction shows the headline rates are plain task counts, not weighted aggregates.

How to maintain it

  • Pin the container base image and per-task Dockerfiles. Scores depend on the exact library versions layered into each task container; a base-image bump can move a metric without any agent change, so treat the NatureGym image set as a versioned unit exactly as for any evaluation harness.
  • Report against a named model-and-harness roster. The results pair a model with a specific harness (Claude Code, Codex CLI, or Gemini CLI); the same model under a different harness is a different configuration, and mixing them silently changes the ranking.
  • Keep the SOTA anchor current. Every g is relative to the source paper's reported SOTA at distillation time; if a task's field advances, the "published SOTA" the agent is measured against is stale, and a surpass may mean the agent beat a now-superseded number rather than the current frontier. Re-anchor tasks when their fields move.
  • Distinguish agent failure from environment or budget exhaustion. A run can fail by wrong method, by hitting the 4-hour wall-clock, or by a container fault; the metric floors non-submission at -1.0, so log the reason per instance rather than letting a timeout land in the agent's competence column.
  • Reconcile "ten" versus "twelve" before quoting a roster size. The abstract's stale "ten frontier agent configurations" is the pre-revision (v1) roster the mechanism analysis still runs on; the main results table has the current twelve rows, added in v2. State which one a number came from, and do not assume the ten-agent subset is a deliberate, curated drop of today's two weakest rows.

Running it in production

The production role is a pre-deployment gate on scientific problem-solving, one tier stricter than artifact completion. The paper's mode analysis gives the checklist its content. First, do not credit matches as discoveries: match-band success is dominated by methodological translation (converting the task into a supervised-prediction problem), which accounts for 45.5% of validated successes, with engineering-driven pathways (supervised proxy, optimization and tuning, engineering pipelines, pretraining or scaling) totaling roughly 82.7% versus about 8.3% domain-reasoned and 9.0% method-aligned; an agent that "solves" a science task may just be recognizing a regression problem.5 Second, weight the failure modes the benchmark actually finds: below-SOTA runs are dominated by wrong method choice (about 45.1%) and insufficient compute budget or time (about 24.4%), not by misunderstanding the task (about 10.1%), which says the bottleneck is scientific judgment and resourcing rather than comprehension.5 Third, budget compute explicitly: the fixed 4-hour, single-GPU envelope is part of the score, so a production gate must decide its own budget rather than assume unlimited runs. Fourth, keep the anti-shortcut posture: web search stays disabled and the ground truth stays hidden, the same evaluation-integrity discipline any real gate needs, or the number measures retrieval instead of discovery.

Failure modes

  • Wrong method choice dominates. The largest failure share (about 45.1% of below-SOTA runs) is the agent choosing an approach that cannot reach SOTA, not one that misreads the task; a config that looks fluent can still pick the wrong scientific tool.5
  • Compute exhaustion is a real failure, not noise. About 24.4% of failures are insufficient budget or time against the 4-hour, single-GPU envelope; a slower or less compute-efficient agent loses tasks it might solve with more runway, so the score partly measures efficiency.5
  • Matching read as discovering. The metric's match band (0 to 0.1) and surpass band (>0.1) are different outcomes; averaging or reporting only "match rate" hides how rarely agents actually improve on SOTA (17.8% at best versus 47.8% matching).4
  • The SOTA==0 edge is undefined. Where a task's SOTA value is 0, the normalizer |m_sota| vanishes and g is undefined; the paper specifies no handling, so any re-implementation or new task must decide this case explicitly rather than divide by zero.3
  • Stale SOTA anchors. A surpass is relative to the source paper's reported number; on a fast-moving task the published SOTA may already be beaten in the literature, so a g>0.1 can overstate genuine novelty. Re-anchor before trusting a discovery claim.
  • Roster and version drift. The abstract's stale "ten" and the table's current "twelve" refer to different rosters from different paper revisions, unreconciled anywhere in the text; a summary written against the wrong roster size, or one that assumes the ten-agent subset drops today's two weakest rows, can misstate what was evaluated. Cite v2's twelve-row table for results.

References

  • Wang, Cheng, Zuo, Zeng, He, Jiang, Yang, Wang, Zhao, Huang, Tian, Yuan, Zhong, Wang, Ding, Zhou, and Zhang, "NatureBench: Can Coding Agents Match the Published SOTA of Nature-Family Papers?" (arXiv 2606.24530): https://arxiv.org/abs/2606.24530
  • NatureBench full text (v2, HTML): https://arxiv.org/html/2606.24530v2
  • NatureBench code and NatureGym pipeline (FrontisAI): https://github.com/FrontisAI/NatureBench
  • NatureBench leaderboard: https://frontisai.github.io/NatureBench/
  • NatureBench dataset (Hugging Face): https://huggingface.co/datasets/FrontisAI/NatureBench

Related: Evaluating agents · Workspace-Bench · AIOpsLab · Cybersecurity agent evaluation · Evaluation integrity and anti-gaming · LLM evaluation harness and eval gate · LLM benchmarks (anatomy and metrics) · LLM judge reliability · Glossary

  1. NatureBench (arXiv 2606.24530), Abstract and Section 1. "A cross-discipline benchmark of 90 tasks distilled from peer-reviewed Nature-family publications, designed to evaluate whether AI coding agents can move beyond reproduction toward discovery on real scientific problems." Six domains: cellular omics, protein biology, biomedical modeling, physical modeling, molecular design, relational reasoning. Publication window 2022-2025. Journals concentrate in Nature Machine Intelligence (36 tasks), Nature Methods (26), Nature Computational Science (16); the remaining 12 tasks (90 minus 78) fall across three other Nature-family journals. Absolute per-domain task counts are shown graphically (Figure 3), not as a published table, so this page does not quote per-domain counts. ↩↩

  2. NatureBench Section 3 and Table 1 (NatureGym). Task package: agent-visible problem/README.md, problem/data_description.md, problem/data/ (inputs only); hidden evaluation/evaluator.py, evaluation/ground_truth/; infrastructure environment/Dockerfile, metadata.json. "Prior benchmarks suffer from environment fragmentation that makes independent re-running fragile"; NatureGym uses a shared base image with pre-installed scientific and ML libraries and layers task-specific dependencies via per-task Dockerfiles. "An information firewall removes the source method from each package, so agents must discover solutions rather than reproduce them." Protocol: web search disabled; isolated NatureBench container with read/write workspace/, 4-hour wall-clock budget, GPU matched to metadata (3 tasks CPU-only, 70 single RTX 3090/4090, 17 single A800). 90 tasks and 333 evaluation instances, mean 3.7 instances per task (median 3). ↩↩↩↩↩↩↩↩

  3. NatureBench Section 4 (Equation 1). g_i = dir_i * (m_i - m_i_sota) / |m_i_sota|, dir_i in {+1, -1} for higher/lower-is-better. "g_i >= 0 means the agent matches or surpasses the published result." Surpass criterion g > 0.1; match criterion g >= 0; reproduce-mode success g >= -0.05; instances with no valid submission receive g_fail = -1.0. Task-level score averages g_i across instances. The paper gives no handling for m_sota = 0, where the normalizer |m_sota| is zero and g is undefined; the executed model raises rather than dividing by zero. ↩↩↩↩

  4. NatureBench Section 5 and Table 4 (arXiv 2606.24530v2). "The strongest agent, Claude Opus 4.7, surpasses the published SOTA (g>0.1) on only 17.8% of tasks and matches it on 47.8%." Table 4 "All" columns (Surpass g>0.1 / Match g>=0), twelve configurations: Claude Opus 4.7 17.8/47.8, GLM-5.2 15.6/41.1, Gemini 3.5 Flash 15.6/37.8, GPT-5.5 14.4/44.4, Claude Opus 4.6 12.2/36.7, MiniMax-M3 11.1/33.3, Qwen 3.7 Max 10.0/28.9, Kimi K2.6 8.9/30.0, GPT-5.4 8.9/27.8, GLM-5.1 7.8/28.9, DeepSeek-V4-Pro 4.4/26.7, MiniMax-M2.7 1.1/13.3. All rates are integer task counts over 90 (17.8% = 16/90, 47.8% = 43/90). No aggregate mean is stated. Claude Opus 4.7 per-domain surpass: Physical 30.8, Cellular 22.6, Molecular 18.2, Protein 12.5, Biomedical 7.1, Relational 0.0 (small per-domain denominators). Reproduce calibration: Claude Opus 4.6 reproduces 30 tasks (g >= -0.05), DeepSeek-V4-Pro reproduces 21. Roster note: the abstract says "ten frontier agent configurations," unchanged since v1, while Table 4 lists twelve rows added by v2 (the v2 arXiv changelog reads "Add results of GLM-5.2 and MinMax-M3"). Section 5's method-pathway analysis reports percentages byte-identical between v1 and v2, indicating it was never rerun on the v2 roster; the ten-agent subset it analyzes is therefore the original v1 roster, including GLM-5.1 and MiniMax-M2.7 and excluding the v2-added GLM-5.2 and MiniMax-M3, not a curated drop of the current two weakest rows (which are MiniMax-M2.7 and DeepSeek-V4-Pro by surpass rate, not MiniMax-M2.7 and GLM-5.1). The paper does not state this reconciliation explicitly. ↩↩↩↩↩↩

  5. NatureBench Section 5 (method-pathway and failure analysis, Figure 7). "Success is driven primarily by methodological translation, where agents convert scientific tasks into familiar supervised-prediction problems, accounting for 45.5% of validated successes, rather than by scientific invention." Success-mode shares among Match-SOTA runs: supervised proxy prediction 45.5%, optimization and tuning 17.6%, engineering pipelines 11.0%, pretraining or model scaling 8.6% (engineering-driven total about 82.7%), versus domain-reasoned about 8.3% and method-aligned about 9.0%. Failure-mode shares among below-SOTA runs: wrong method choice about 45.1%, insufficient budget or time about 24.4%, understanding or strategy about 10.1%, other execution about 4.3%. Abstract summary: "Failures are dominated by wrong method choice and insufficient compute budget, not by task misunderstanding." ↩↩↩↩↩