Markdown

Agentic paper replication (evidence-gated reproduction)¶

Scope: a coding-agent workflow for reproducing the computational claims of a scientific paper from paper materials alone (no author code), and the harness-engineering mechanism that stops the agent from marking a claim "done" on the strength of its own final message. Proposed as Paper-replication in Hans and Bilionis, "Coding-agents can replicate scientific machine learning papers" (arXiv 2607.02134). This is the evidence-gated-completion pattern applied specifically to scientific-claim reproduction, distinct from general coding evals like SWE-bench or file-dependency benchmarks like Workspace-Bench.

The stdlib-only block below is an executed minimal model of the paper's completion rule, not the paper's implementation. The released tooling is linked in References. This is one paper with one coding-agent configuration (Codex, GPT-5.4, Extra High reasoning), no ablation against unstructured prompting, and a four-paper corpus. The results demonstrate feasibility; they do not establish a general benchmark advantage.

flowchart LR
  PAPER["Paper materials only<br/>(LaTeX source, figures, tables, no author code)"] --> TARGETS["Record targets<br/>(reproduction matrix)"]
  TARGETS --> METHOD["Reconstruct method<br/>(specification files)"]
  METHOD --> RUN["Run experiments<br/>(run recorder + provenance)"]
  RUN --> COMPARE["Compare to paper claim<br/>(target-specific acceptance rule)"]
  COMPARE --> EVIDENCE["Evidence bundle:<br/>run + provenance + comparison + report coverage"]
  EVIDENCE -->|"all four present"| MATCHED["Target: MATCHED"]
  EVIDENCE -->|"any missing"| UNMATCHED["Target: UNMATCHED"]
  MATCHED --> GATE{"Completion gate:<br/>all targets MATCHED,<br/>no active target,<br/>report PDF exists"}
  GATE -->|"pass"| DONE["Workspace: complete"]
  GATE -->|"fail"| RUN

What it is¶

Paper-replication is a coding-agent skill (instruction files plus workspace utilities, scripts/paper_replication.py) that turns "replicate this paper" into a target-level evidence contract. The agent inspects the paper's LaTeX source, figures, tables, and datasets, and records every computational claim it will reproduce as a target in a reproduction matrix. Each target only becomes MATCHED once its evidence bundle exists: a successful, recorded run; a provenance link tying the generated output to the agent's reconstruction of the paper's method (code, config, seed, and the paper passages that justify it); a comparison against the paper's claim under a target-specific acceptance rule; and coverage of that evidence in the final replication report. A completion gate then checks the whole workspace: every recorded target must be MATCHED, no target may remain active, and the rendered report PDF must exist. None of this depends on what the agent says in its last chat message.

Why use it¶

Closes the "agent says done" gap. Direct prompting can let an agent stop after partial work or treat progress narration as evidence. The paper uses Goodhart's law as a conceptual warning about optimizing a completion signal; Manheim and Garrabrant do not empirically test coding-agent completion reports.
An output artifact alone is not evidence. A generated figure or table that merely looks right does not satisfy a target: the workspace also has to show a successful run, provenance back to a reconstructed method, and a report location. Hash checks separately detect an agent reusing a paper-provided figure or table as if it were a generated result.
Persistent, resumable state. The reproduction matrix, task ledger, specification files, run records, and provenance records live in the workspace, not the chat transcript, so a long-running replication can survive interruptions and resume without losing track of which target is active.
Demonstrated end to end. Across 12 independent runs on 4 scientific machine learning papers, all 12 workspaces reached the completion gate and all 158 recorded targets were matched with report coverage.

When to use it (and when not)¶

Use it when a coding agent needs to reproduce specific, checkable computational claims from a paper (or similar spec document) and you need the completion signal to be trustworthy enough to act on, not just a chat message that says "done."
Use the general pattern ("target with a recorded evidence bundle, gate on workspace state") for any long-horizon agent task where the failure mode is the agent under-delivering while claiming success, well beyond paper replication: this is the same discipline the evaluation and anti-gaming pages call for in agent evals generally.
Do not treat a MATCHED target as proof of exact numerical reproduction. The paper is explicit that MATCHED reflects the acceptance rule recorded for that target, not a universal fidelity metric; two of the paper's own 39 independently re-checked scalar anchors fall outside a fixed external threshold while still counting as MATCHED under their own workspace's recorded rule (see Results).
Do not assume this generalizes past scientific machine learning code. The evidence, targets, and thresholds are all specific to reproducing computational claims (differentiable solvers, PINNs, sparse system identification); the mechanism (evidence-bundle-gated completion) is general, the specific acceptance-rule vocabulary is not.
Do not use it where author code and data are available and permitted. Paper-replication is built for the harder no-author-code setting; if a reproduction package exists, a simpler rerun-and-diff workflow (as in CORE-Bench) is the right tool.

Architecture¶

flowchart TB
  subgraph Workspace records
    MANIFEST["Manifest: paper source hash, run rules, compute env"]
    MATRIX["Reproduction matrix: one row per target, current status"]
    LEDGER["Task ledger: the one ACTIVE target, open questions"]
    SPEC["Specification files: reconstructed equations, algorithms, assumptions"]
    RUNREC["Run record: command, working dir, success, output hashes"]
    PROV["Provenance record: output -> implementation, config, seed, paper passage"]
    REPORT["Replication report (report/main.pdf)"]
  end
  subgraph Validation checks
    VSPEC["V_spec: specification completeness"]
    VPROG["V_progress: ledger/matrix consistency, provenance present, copied-asset hash check"]
    VREP["V_report: matched targets appear in the report"]
  end
  MANIFEST --> MATRIX --> LEDGER --> SPEC --> RUNREC --> PROV --> REPORT
  SPEC --> VSPEC
  LEDGER --> VPROG
  MATRIX --> VPROG
  PROV --> VPROG
  REPORT --> VREP
  VSPEC --> GATE["Completion gate (Eq. 3)"]
  VPROG --> GATE
  VREP --> GATE
  GATE -->|"all targets MATCHED, no ACTIVE target, report exists"| COMPLETE["Workspace: complete"]

Core mechanism (runnable): the completion gate¶

This stdlib-only block models both levels of the rule: target evidence must pass before a target becomes MATCHED, and the workspace gate requires V_spec, V_progress, and V_report in addition to matched targets, no active target, and a report PDF. Assertions cover missing provenance, missing report coverage, numeric boundaries, every validator input, an unmatched target, an active target, and a missing report. Run: python3 completion_gate.py.

# completion_gate.py: runnable minimal model of the two-level evidence contract.
# The target-level check requires run, provenance, comparison, and report coverage.
# The workspace-level check includes all terms shown in the paper's Eq. 3.
# This is not the released paper_replication.py implementation.


PLANNED, ACTIVE, MATCHED, UNMATCHED = "PLANNED", "ACTIVE", "MATCHED", "UNMATCHED"

def evaluate_target(run_success, provenance_linked, discrepancy, tolerance, report_covered):
    # Evidence bundle E_j = (y_hat_j, R_j, P_j, C_j, G_j): run record, provenance,
    # comparison, report coverage. All four must hold; a numeric pass alone is not enough.
    comparison_pass = discrepancy <= tolerance          # C_j: acceptance rule for a numeric target
    if run_success and provenance_linked and comparison_pass and report_covered:
        return MATCHED
    return UNMATCHED

def completion_gate(target_statuses, active_target, report_pdf_exists,
                    spec_valid, progress_valid, report_valid):
    # Eq. 3 includes the three validators as well as target and artifact state.
    all_matched = all(s == MATCHED for s in target_statuses)
    no_active = active_target is None
    return all((spec_valid, progress_valid, report_valid,
                all_matched, no_active, report_pdf_exists))

# 1) Happy path: PINN-I solution relative L2 error, threshold 1e-2 (paper's Table 1 rule).
status = evaluate_target(run_success=True, provenance_linked=True,
                          discrepancy=0.0073, tolerance=1e-2, report_covered=True)
assert status == MATCHED, status

# 2) Adversarial (load-bearing): a numerically-perfect output with NO provenance link
#    must not be accepted, even though the comparison alone would pass. This is the
#    exact failure mode Paper-replication is built to close (a copied paper figure or
#    a substitute method that "looks right").
status_no_provenance = evaluate_target(run_success=True, provenance_linked=False,
                                        discrepancy=0.0001, tolerance=1e-2, report_covered=True)
assert status_no_provenance == UNMATCHED, "unprovenanced output must not be accepted as evidence"

# 3) Adversarial: report coverage missing (result never lands in the replication report)
#    must also block MATCHED, per the report-coverage check.
status_no_report = evaluate_target(run_success=True, provenance_linked=True,
                                    discrepancy=0.0001, tolerance=1e-2, report_covered=False)
assert status_no_report == UNMATCHED, "missing report coverage must not be accepted"

# 4) Boundary: exactly at tolerance passes (<=), just above fails. Uses the paper's own
#    PINN-II coefficient-error convention (10 percent, an analysis convention the authors
#    state explicitly, not a claim that data noise and coefficient error are equivalent).
at_bound = evaluate_target(True, True, discrepancy=0.10, tolerance=0.10, report_covered=True)
past_bound = evaluate_target(True, True, discrepancy=0.1001, tolerance=0.10, report_covered=True)
assert at_bound == MATCHED and past_bound == UNMATCHED, (at_bound, past_bound)

# 5) All 158 targets, all validators, and all terminal-state checks must pass.
targets_158 = [MATCHED] * 158
validators = {"spec_valid": True, "progress_valid": True, "report_valid": True}
assert completion_gate(targets_158, None, True, **validators) is True

# 6) Unmatched, active, and missing-report states each block completion.
targets_one_unmatched = [MATCHED] * 157 + [UNMATCHED]
assert completion_gate(targets_one_unmatched, None, True, **validators) is False
assert completion_gate(targets_158, "t042", True, **validators) is False
assert completion_gate(targets_158, None, False, **validators) is False

# 7) Each Eq. 3 validator is independently load-bearing.
validator_failures_blocked = True
for key in validators:
    invalid = {**validators, key: False}
    blocked = not completion_gate(targets_158, None, True, **invalid)
    validator_failures_blocked = validator_failures_blocked and blocked
    assert blocked, f"{key} must block completion"

print("Paper-replication completion-gate core mechanism: PASS")
print("happy-path MATCHED:", status, "| no-provenance blocked:", status_no_provenance,
      "| no-report blocked:", status_no_report)
print("boundary at 10%:", at_bound, "| boundary at 10.01%:", past_bound)
valid_result = completion_gate(targets_158, None, True, **validators)
unmatched_result = completion_gate(targets_one_unmatched, None, True, **validators)
print("158/158 gate:", valid_result,
      "| 1 unmatched blocks gate:", unmatched_result)
print("validator failure blocks gate:", validator_failures_blocked)

Executed output:

Paper-replication completion-gate core mechanism: PASS
happy-path MATCHED: MATCHED | no-provenance blocked: UNMATCHED | no-report blocked: UNMATCHED
boundary at 10%: MATCHED | boundary at 10.01%: UNMATCHED
158/158 gate: True | 1 unmatched blocks gate: False
validator failure blocks gate: True

How it works¶

Four target types get different acceptance rules, because not every scientific claim is a single number: a numeric target (a reported relative error, an estimated coefficient) is judged by a discrepancy metric against a recorded tolerance; a distributional target (posterior shape, bimodality, sampling behavior) is judged by distributional properties rather than sample-for-sample equality; a structural target (sparsity pattern, phase-portrait geometry, attractor structure) is judged by whether the claimed structural property holds; a visual target is judged by a recorded comparison between the reference and candidate output, used only when the claim itself is visual, since stochastic trajectories and convergence curves can support the same claim without pixel-identical agreement. Whichever rule applies, the agent records it in the target before judging the output, so it cannot change the judgment rule after seeing a failed comparison.

Separately, every substantive execution passes through a run recorder that stores the command, working directory, timestamps, success/failure, and output hashes; failed and later-superseded runs stay in the record rather than being overwritten, preserving the trial-and-correction path. Paper-provided assets (figures, source files, rendered pages) are hashed and kept in a directory separate from agent-generated outputs, so a later check can catch an output that reuses paper-provided material by path or hash. The paper is candid that hash checks do not catch every transformed copy, and that the provenance and comparison records are the backstop.

How to use it¶

Give the agent the paper materials, not the paper's code. The reported setup deliberately withholds author code; the agent infers the method from the LaTeX source, figures, tables, and referenced datasets, treating any missing implementation detail (seed, tolerance, preprocessing, plotting convention) as a hypothesis it records and tests rather than something to guess silently.
Let the agent pick its own target decomposition, but hold it to the evidence contract. The reported runs show real variation in how finely a paper is split into targets: PINN-I and SINDy were decomposed identically across all three runs of each, while PIFT ranged from 8 to 25 targets and PINN-II from 9 to 15. That variation is fine; every target, however many, still has to pass the same evidence bundle.
Use a follow-up prompt queue for long-running replications, not a bigger single prompt. The reported protocol reopens the same workspace on a fixed queue of follow-up prompts ("continue until all targets are matched and the completion gate passes") so a long-horizon run can resume from workspace state rather than restarting from a summarized transcript, the same discipline covered in context and memory.
Budget for real wall-clock time. Reported elapsed replication time (posterior median from the effort model) ranged from 1.9 hours (SINDy) to 6.9 hours (PINN-II) per run, with individual runs up to 13.0 hours; this is a multi-hour agent task, not a single-turn one.

How to integrate with it¶

As a coding-agent skill. The reference implementation ships as a Codex and Claude Code skill: an instruction layer (SKILL.md plus per-agent prompt adaptations) and workspace utilities (scripts/paper_replication.py) that create and check the manifest, reproduction matrix, task ledger, specification files, and report template. The instruction layer carries no paper-specific science; it is the workflow contract, reusable across papers.
As a harness-engineering pattern, not a prompt. The paper explicitly frames this as harness engineering (citing Lopopolo, 2026, the same term used in this KB's loop engineering and harness architecture pages): the fix for an unreliable prompt is to change the environment the agent works in (persistent files, external validation) rather than to word the prompt more carefully.
Compute environment is pluggable. The reported runs mixed local execution (a MacBook Pro, M4 Max, 128 GB) with a cluster-execution skill routing heavier jobs to shared CPU/GPU nodes; the run-record and provenance mechanism is the same regardless of where a given execution actually ran.
The evidence-bundle-gated-completion idea generalizes past paper replication. Any long-horizon agent task where "the agent says it's done" is not a trustworthy signal (see evaluating agents and evaluation integrity) can borrow the shape: decompose the task into targets, require a provenance-linked, externally-checked evidence bundle per target, and gate completion on workspace state.

How to run it in production¶

Treat MATCHED as "satisfied its own recorded acceptance rule," not "numerically exact." In the reported case study, of 39 independently re-checked scalar anchor-run observations, 37 fell inside a fixed paper-anchored threshold and 2 did not (a Schrödinger PINN-I run at 4.8e-2 against a 1 percent threshold, and a Navier-Stokes PINN-II run at 16.4 percent against a 10 percent threshold), yet both targets were still recorded MATCHED because the workspace's own acceptance rule for that target accepted them. If you need a stricter external audit, re-check scalar claims against a fixed threshold as a separate pass, the way the paper's own case-study analysis does, rather than trusting MATCHED alone.
Watch the run-to-run headroom and residual-scale numbers, not just the pass/fail count. The reported headroom model gives posterior-median headroom (log10 units) of 0.51 for PINN-I, 1.75 for PINN-II, and 0.42 for SINDy (roughly 3.2x, 57x, and 2.6x inside threshold on average), but the run-residual scale is largest for PINN-II (posterior median 1.16, about a 14x spread across reruns): completion being stable across runs does not mean fidelity is stable across runs.
Expect and record correction work; do not treat it as failure. Across the reported corpus, 25 of the tracked executions were later superseded by correction work before final evidence was accepted (21 of those in the two PINN papers). The paper's framing is that this trial-and-correction path is part of the evidence, not overhead to hide.
Judgment variation is real: two completed workspaces can classify the same claim differently. The reported same-acceptance-rule-type agreement across repeated runs of the same paper ranged from 0.95 (SINDy) down to 0.46 (PINN-II); if you are aggregating MATCHED status across runs or agents, do not assume they used the same kind of evidence for the same claim.

How to maintain it¶

Re-run the completion-gate assertions on any change to the evidence-bundle logic. The stdlib-only block above checks target evidence and every workspace-level validator; a regression that bypasses any term reopens the failure mode the gate is designed to close.
Keep the target-type-specific acceptance rules explicit and versioned. Numeric tolerances (1e-2, 10 percent, 1e-3 in the reported study) are per-paper analysis conventions, not universal constants; do not hard-code a single tolerance across target types.
Preserve superseded run records rather than overwriting them. The trial-and-correction path (25 superseded executions in the reported corpus) is part of what makes the workspace auditable; a maintenance change that prunes "failed" runs to save space destroys the evidence the completion gate is supposed to produce.
Re-validate hash-based copied-asset checks after any change to the workspace directory layout. The separation between paper-provided assets and agent-generated outputs is what the hash check relies on; the paper is explicit that this check does not catch every transformed copy, so keep the provenance and comparison checks as the backstop, not the hash check alone.

Results¶

Reported on 12 independent runs (3 per paper) across 4 scientific machine learning papers: physics-informed information field theory (PIFT), two physics-informed neural network papers (PINN-I: forward solutions, PINN-II: coefficient discovery), and sparse identification of nonlinear dynamics (SINDy). Coding agent: Codex with GPT-5.4 at the "Extra High" reasoning setting, author code disallowed.

Paper	Runs	Targets/run	All matched	Elapsed time, median [95% CI] (h)	Superseded executions
PIFT	3	8, 8, 25	yes	2.2 [1.1, 4.4]	3
PINN-I	3	8, 8, 8	yes	5.0 [2.5, 9.9]	11
PINN-II	3	9, 9, 15	yes	6.9 [3.0, 13.4]	10
SINDy	3	20, 20, 20	yes	1.9 [1.0, 4.3]	1

All 12 workspaces reached the completion gate; all 158 recorded targets were MATCHED with report coverage. Of 39 independently re-checked scalar anchor-run observations (13 anchors x 3 runs), 37 fell inside a fixed paper-anchored threshold and 2 did not, while remaining MATCHED under their own workspace's acceptance rule. Same-acceptance-rule-type agreement across repeated runs of a paper ranged from 19/20 (SINDy) to 5/11 (PINN-II).

Failure modes¶

Mistaking MATCHED for exact numerical reproduction. It means the recorded, claim-specific acceptance rule was satisfied, not that the reproduced value equals the paper's printed digit for digit; the paper's own PINN-II clean-coefficient errors varied from 7.3 percent to 0.014 percent across reruns, all still under the 10 percent convention used.
Trusting hash checks alone against copied material. They compare paths and hashes against indexed paper assets and rendered pages, and the authors state plainly that this does not rule out every transformed copy or adversarial reuse; provenance and comparison records are the actual backstop.
Judging a workflow's effect from this study alone. There is no reported ablation against unstructured prompting, so the numbers characterize what Paper-replication produces, not how much better it is than prompting an agent without the skill.
Small-corpus overreach. Four papers and three runs each is enough to see within-paper run-to-run variation, but the authors caution the corpus is too small to draw strong between-paper conclusions or to rank paper difficulty from elapsed time or correction-work counts.
Treating scalar thresholds as ground truth. The numeric tolerances used in the case-study re-analysis (1e-2, 10 percent, 1e-3) are the authors' explicit analysis conventions derived from each paper's reported accuracy scale; a different defensible threshold would shift how many anchors count as "inside," without changing the underlying reproduced values.
No scalar anchor for non-numeric claims. PIFT contributes zero scalar anchors because its claims (posterior collapse, bimodality, selective identifiability) are distributional and structural; do not force a headroom-style analysis onto a target type it does not fit.

References¶

Hans, A., Bilionis, I., Coding-agents can replicate scientific machine learning papers: https://arxiv.org/abs/2607.02134 (code, prompts, and case-study workspaces: https://github.com/PredictiveScienceLab/paper-replication-paper)
Lopopolo, R., Harness engineering: leveraging Codex in an agent-first world: https://openai.com/index/harness-engineering/
Starace et al., PaperBench: Evaluating AI's Ability to Replicate AI Research: https://arxiv.org/abs/2504.01848
Siegel et al., CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark: https://arxiv.org/abs/2409.11363
Seo et al., PaperCoder / Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning: https://arxiv.org/abs/2504.17192
Xiang et al., SciReplicate-Bench: Benchmarking LLMs in Agent-Driven Algorithmic Reproduction from Research Papers: https://arxiv.org/abs/2504.00255
Manheim, D., Garrabrant, S., Categorizing Variants of Goodhart's Law: https://arxiv.org/abs/1803.04585