Copy for agents Markdown

Agentic paper replication (evidence-gated reproduction)

Scope: a coding-agent workflow for reproducing the computational claims of a scientific paper from paper materials alone (no author code), and the harness-engineering mechanism that stops the agent from marking a claim "done" on the strength of its own final message. Proposed as Paper-replication in Hans and Bilionis, "Coding-agents can replicate scientific machine learning papers" (arXiv 2607.02134). This is the evidence-gated-completion pattern applied specifically to scientific-claim reproduction, distinct from general coding evals like SWE-bench or file-dependency benchmarks like Workspace-Bench.

The stdlib-only block below is an executed minimal model of the paper's completion rule, not the paper's implementation. This is one paper with one coding-agent configuration (Codex, GPT-5.4, Extra High reasoning), no ablation against unstructured prompting, and a four-paper corpus. The results demonstrate feasibility; they do not establish a general benchmark advantage.

Independently verified/executed here: the released tooling itself, cloned from github.com/PredictiveScienceLab/paper-replication-paper at commit e030a7b5dc625acb7cfc1b9b5630161b7a4a1ed2 (2026-07-05, Apache 2.0), was installed and driven end to end through its real, unmodified paper_replication.py CLI, not a paraphrase of it. A minimal two-target synthetic paper (a numeric Monte Carlo pi estimate and a structural sparse-identification claim) was bootstrapped, spec'd, run, and pushed through validate-spec, validate-progress, validate-report, and validate-completion until the real completion gate returned ok: true, with a compiled report/main.pdf. Two adversarial cases were run against the real validators, not simulated: a provenance record whose implementation summary contained a forbidden pattern-matching marker was rejected by register-target-artifact, and a numeric-equivalence target given only a visual (SSIM) metric was rejected by validate-progress. The structural target's first attempt (STLSQ threshold 0.05) genuinely failed to recover the paper's claimed sparsity pattern; the correction (threshold 0.1, documented as a superseded run, not deleted) is the real trial-and-correction path the paper describes, reproduced here rather than asserted. The shipped test_paper_replication_unit.py suite was also run unmodified: 39 of 41 tests pass; the 2 failures are both due to the sibling cluster-slurm skill not being installed in this sandbox, not a defect in this skill. Full commands, schemas, and output are in "How to run the released harness end to end" below. All of this used paper-replication only; Codex, GPT-5.4, and the case-study corpus in Results below were not re-run, and remain the paper's own reported numbers.

flowchart LR
  PAPER["Paper materials only<br/>(LaTeX source, figures, tables, no author code)"] --> TARGETS["Record targets<br/>(reproduction matrix)"]
  TARGETS --> METHOD["Reconstruct method<br/>(specification files)"]
  METHOD --> RUN["Run experiments<br/>(run recorder + provenance)"]
  RUN --> COMPARE["Compare to paper claim<br/>(target-specific acceptance rule)"]
  COMPARE --> EVIDENCE["Evidence bundle:<br/>run + provenance + comparison + report coverage"]
  EVIDENCE -->|"all four present"| MATCHED["Target: MATCHED"]
  EVIDENCE -->|"any missing"| UNMATCHED["Target: UNMATCHED"]
  MATCHED --> GATE{"Completion gate:<br/>all targets MATCHED,<br/>no active target,<br/>report PDF exists"}
  GATE -->|"pass"| DONE["Workspace: complete"]
  GATE -->|"fail"| RUN

What it is

Paper-replication is a coding-agent skill (instruction files plus workspace utilities, scripts/paper_replication.py) that turns "replicate this paper" into a target-level evidence contract. The agent inspects the paper's LaTeX source, figures, tables, and datasets, and records every computational claim it will reproduce as a target in a reproduction matrix. Each target only becomes MATCHED once its evidence bundle exists: a successful, recorded run; a provenance link tying the generated output to the agent's reconstruction of the paper's method (code, config, seed, and the paper passages that justify it); a comparison against the paper's claim under a target-specific acceptance rule; and coverage of that evidence in the final replication report. A completion gate then checks the whole workspace: every recorded target must be MATCHED, no target may remain active, and the rendered report PDF must exist. None of this depends on what the agent says in its last chat message.

Why use it

• Closes the "agent says done" gap. Direct prompting can let an agent stop after partial work or treat progress narration as evidence. The paper uses Goodhart's law as a conceptual warning about optimizing a completion signal; Manheim and Garrabrant do not empirically test coding-agent completion reports.
• An output artifact alone is not evidence. A generated figure or table that merely looks right does not satisfy a target: the workspace also has to show a successful run, provenance back to a reconstructed method, and a report location. Hash checks separately detect an agent reusing a paper-provided figure or table as if it were a generated result.
• Persistent, resumable state. The reproduction matrix, task ledger, specification files, run records, and provenance records live in the workspace, not the chat transcript, so a long-running replication can survive interruptions and resume without losing track of which target is active.
• Demonstrated end to end. Across 12 independent runs on 4 scientific machine learning papers, all 12 workspaces reached the completion gate and all 158 recorded targets were matched with report coverage.

When to use it (and when not)

• Use it when a coding agent needs to reproduce specific, checkable computational claims from a paper (or similar spec document) and you need the completion signal to be trustworthy enough to act on, not just a chat message that says "done."
• Use the general pattern ("target with a recorded evidence bundle, gate on workspace state") for any long-horizon agent task where the failure mode is the agent under-delivering while claiming success, well beyond paper replication: this is the same discipline the evaluation and anti-gaming pages call for in agent evals generally.
• Do not treat a MATCHED target as proof of exact numerical reproduction. The paper is explicit that MATCHED reflects the acceptance rule recorded for that target, not a universal fidelity metric; two of the paper's own 39 independently re-checked scalar anchors fall outside a fixed external threshold while still counting as MATCHED under their own workspace's recorded rule (see Results).
• Do not assume this generalizes past scientific machine learning code. The evidence, targets, and thresholds are all specific to reproducing computational claims (differentiable solvers, PINNs, sparse system identification); the mechanism (evidence-bundle-gated completion) is general, the specific acceptance-rule vocabulary is not.
• Do not use it where author code and data are available and permitted. Paper-replication is built for the harder no-author-code setting; if a reproduction package exists, a simpler rerun-and-diff workflow (as in CORE-Bench) is the right tool.

Architecture

flowchart TB
  subgraph Workspace records
    MANIFEST["Manifest: paper source hash, run rules, compute env"]
    MATRIX["Reproduction matrix: one row per target, current status"]
    LEDGER["Task ledger: the one ACTIVE target, open questions"]
    SPEC["Specification files: reconstructed equations, algorithms, assumptions"]
    RUNREC["Run record: command, working dir, success, output hashes"]
    PROV["Provenance record: output -> implementation, config, seed, paper passage"]
    REPORT["Replication report (report/main.pdf)"]
  end
  subgraph Validation checks
    VSPEC["V_spec: specification completeness"]
    VPROG["V_progress: ledger/matrix consistency, provenance present, copied-asset hash check"]
    VREP["V_report: matched targets appear in the report"]
  end
  MANIFEST --> MATRIX --> LEDGER --> SPEC --> RUNREC --> PROV --> REPORT
  SPEC --> VSPEC
  LEDGER --> VPROG
  MATRIX --> VPROG
  PROV --> VPROG
  REPORT --> VREP
  VSPEC --> GATE["Completion gate (Eq. 3)"]
  VPROG --> GATE
  VREP --> GATE
  GATE -->|"all targets MATCHED, no ACTIVE target, report exists"| COMPLETE["Workspace: complete"]

Core mechanism (runnable): the completion gate

This stdlib-only block models both levels of the rule: target evidence must pass before a target becomes MATCHED, and the workspace gate requires V_spec, V_progress, and V_report in addition to matched targets, no active target, and a report PDF. Assertions cover missing provenance, missing report coverage, numeric boundaries, every validator input, an unmatched target, an active target, and a missing report. Run: python3 completion_gate.py.

# completion_gate.py: runnable minimal model of the two-level evidence contract.
# The target-level check requires run, provenance, comparison, and report coverage.
# The workspace-level check includes all terms shown in the paper's Eq. 3.
# This is not the released paper_replication.py implementation.


PLANNED, ACTIVE, MATCHED, UNMATCHED = "PLANNED", "ACTIVE", "MATCHED", "UNMATCHED"

def evaluate_target(run_success, provenance_linked, discrepancy, tolerance, report_covered):
    # Evidence bundle E_j = (y_hat_j, R_j, P_j, C_j, G_j): run record, provenance,
    # comparison, report coverage. All four must hold; a numeric pass alone is not enough.
    comparison_pass = discrepancy <= tolerance          # C_j: acceptance rule for a numeric target
    if run_success and provenance_linked and comparison_pass and report_covered:
        return MATCHED
    return UNMATCHED

def completion_gate(target_statuses, active_target, report_pdf_exists,
                    spec_valid, progress_valid, report_valid):
    # Eq. 3 includes the three validators as well as target and artifact state.
    all_matched = all(s == MATCHED for s in target_statuses)
    no_active = active_target is None
    return all((spec_valid, progress_valid, report_valid,
                all_matched, no_active, report_pdf_exists))

# 1) Happy path: PINN-I solution relative L2 error, threshold 1e-2 (paper's Table 1 rule).
status = evaluate_target(run_success=True, provenance_linked=True,
                          discrepancy=0.0073, tolerance=1e-2, report_covered=True)
assert status == MATCHED, status

# 2) Adversarial (load-bearing): a numerically-perfect output with NO provenance link
#    must not be accepted, even though the comparison alone would pass. This is the
#    exact failure mode Paper-replication is built to close (a copied paper figure or
#    a substitute method that "looks right").
status_no_provenance = evaluate_target(run_success=True, provenance_linked=False,
                                        discrepancy=0.0001, tolerance=1e-2, report_covered=True)
assert status_no_provenance == UNMATCHED, "unprovenanced output must not be accepted as evidence"

# 3) Adversarial: report coverage missing (result never lands in the replication report)
#    must also block MATCHED, per the report-coverage check.
status_no_report = evaluate_target(run_success=True, provenance_linked=True,
                                    discrepancy=0.0001, tolerance=1e-2, report_covered=False)
assert status_no_report == UNMATCHED, "missing report coverage must not be accepted"

# 4) Boundary: exactly at tolerance passes (<=), just above fails. Uses the paper's own
#    PINN-II coefficient-error convention (10 percent, an analysis convention the authors
#    state explicitly, not a claim that data noise and coefficient error are equivalent).
at_bound = evaluate_target(True, True, discrepancy=0.10, tolerance=0.10, report_covered=True)
past_bound = evaluate_target(True, True, discrepancy=0.1001, tolerance=0.10, report_covered=True)
assert at_bound == MATCHED and past_bound == UNMATCHED, (at_bound, past_bound)

# 5) All 158 targets, all validators, and all terminal-state checks must pass.
targets_158 = [MATCHED] * 158
validators = {"spec_valid": True, "progress_valid": True, "report_valid": True}
assert completion_gate(targets_158, None, True, **validators) is True

# 6) Unmatched, active, and missing-report states each block completion.
targets_one_unmatched = [MATCHED] * 157 + [UNMATCHED]
assert completion_gate(targets_one_unmatched, None, True, **validators) is False
assert completion_gate(targets_158, "t042", True, **validators) is False
assert completion_gate(targets_158, None, False, **validators) is False

# 7) Each Eq. 3 validator is independently load-bearing.
validator_failures_blocked = True
for key in validators:
    invalid = {**validators, key: False}
    blocked = not completion_gate(targets_158, None, True, **invalid)
    validator_failures_blocked = validator_failures_blocked and blocked
    assert blocked, f"{key} must block completion"

print("Paper-replication completion-gate core mechanism: PASS")
print("happy-path MATCHED:", status, "| no-provenance blocked:", status_no_provenance,
      "| no-report blocked:", status_no_report)
print("boundary at 10%:", at_bound, "| boundary at 10.01%:", past_bound)
valid_result = completion_gate(targets_158, None, True, **validators)
unmatched_result = completion_gate(targets_one_unmatched, None, True, **validators)
print("158/158 gate:", valid_result,
      "| 1 unmatched blocks gate:", unmatched_result)
print("validator failure blocks gate:", validator_failures_blocked)

Executed output:

Paper-replication completion-gate core mechanism: PASS
happy-path MATCHED: MATCHED | no-provenance blocked: UNMATCHED | no-report blocked: UNMATCHED
boundary at 10%: MATCHED | boundary at 10.01%: UNMATCHED
158/158 gate: True | 1 unmatched blocks gate: False
validator failure blocks gate: True

How it works

Four target types get different acceptance rules, because not every scientific claim is a single number: a numeric target (a reported relative error, an estimated coefficient) is judged by a discrepancy metric against a recorded tolerance; a distributional target (posterior shape, bimodality, sampling behavior) is judged by distributional properties rather than sample-for-sample equality; a structural target (sparsity pattern, phase-portrait geometry, attractor structure) is judged by whether the claimed structural property holds; a visual target is judged by a recorded comparison between the reference and candidate output, used only when the claim itself is visual, since stochastic trajectories and convergence curves can support the same claim without pixel-identical agreement. Whichever rule applies, the agent records it in the target before judging the output, so it cannot change the judgment rule after seeing a failed comparison.

Separately, every substantive execution passes through a run recorder that stores the command, working directory, timestamps, success/failure, and output hashes; failed and later-superseded runs stay in the record rather than being overwritten, preserving the trial-and-correction path. Paper-provided assets (figures, source files, rendered pages) are hashed and kept in a directory separate from agent-generated outputs, so a later check can catch an output that reuses paper-provided material by path or hash. The paper is candid that hash checks do not catch every transformed copy, and that the provenance and comparison records are the backstop.

How to use it

• Give the agent the paper materials, not the paper's code. The reported setup deliberately withholds author code; the agent infers the method from the LaTeX source, figures, tables, and referenced datasets, treating any missing implementation detail (seed, tolerance, preprocessing, plotting convention) as a hypothesis it records and tests rather than something to guess silently.
• Let the agent pick its own target decomposition, but hold it to the evidence contract. The reported runs show real variation in how finely a paper is split into targets: PINN-I and SINDy were decomposed identically across all three runs of each, while PIFT ranged from 8 to 25 targets and PINN-II from 9 to 15. That variation is fine; every target, however many, still has to pass the same evidence bundle.
• Use a follow-up prompt queue for long-running replications, not a bigger single prompt. The reported protocol reopens the same workspace on a fixed queue of follow-up prompts ("continue until all targets are matched and the completion gate passes") so a long-horizon run can resume from workspace state rather than restarting from a summarized transcript, the same discipline covered in context and memory.
• Budget for real wall-clock time. Reported elapsed replication time (posterior median from the effort model) ranged from 1.9 hours (SINDy) to 6.9 hours (PINN-II) per run, with individual runs up to 13.0 hours; this is a multi-hour agent task, not a single-turn one.

How to integrate with it

• As a coding-agent skill. The reference implementation ships as a Codex and Claude Code skill: an instruction layer (SKILL.md plus per-agent prompt adaptations) and workspace utilities (scripts/paper_replication.py) that create and check the manifest, reproduction matrix, task ledger, specification files, and report template. The instruction layer carries no paper-specific science; it is the workflow contract, reusable across papers.
• As a harness-engineering pattern, not a prompt. The paper explicitly frames this as harness engineering (citing Lopopolo, 2026, the same term used in this KB's loop engineering and harness architecture pages): the fix for an unreliable prompt is to change the environment the agent works in (persistent files, external validation) rather than to word the prompt more carefully.
• Compute environment is pluggable. The reported runs mixed local execution (a MacBook Pro, M4 Max, 128 GB) with a cluster-execution skill routing heavier jobs to shared CPU/GPU nodes; the run-record and provenance mechanism is the same regardless of where a given execution actually ran.
• The evidence-bundle-gated-completion idea generalizes past paper replication. Any long-horizon agent task where "the agent says it's done" is not a trustworthy signal (see evaluating agents and evaluation integrity) can borrow the shape: decompose the task into targets, require a provenance-linked, externally-checked evidence bundle per target, and gate completion on workspace state.

How to run the released harness end to end

Install and CLI surface

git clone https://github.com/PredictiveScienceLab/paper-replication-paper.git
cd paper-replication-paper
git checkout e030a7b5dc625acb7cfc1b9b5630161b7a4a1ed2   # 2026-07-05, Apache 2.0
cd skill/codex/paper-replication   # the real script lives here, not at repo root

The repository ships the same harness twice, once per agent (skill/codex/paper-replication/ and skill/claude-code/paper-replication/; the two paper_replication.py files differ slightly per agent conventions but expose the same CLI). It is stdlib-only Python: no pip install is needed to run it. There is no top-level scripts/ directory in this repository; every command below assumes the cd above, into skill/codex/paper-replication/, where scripts/paper_replication.py actually exists (caught on re-review: an earlier revision of this page showed python3 scripts/paper_replication.py run from the repo root instead, which fails with a real, reproduced python3: can't open file '.../scripts/paper_replication.py': [Errno 2] No such file or directory, exit code 2). The CLI is self-documenting:

$ python3 scripts/paper_replication.py --help
usage: paper_replication.py [-h]
                            {bootstrap,sync-harness,inspect-paper,status,validate-spec,validate-progress,
                             validate-completion,validate-report,build-paper-pdf,render-paper-pages,
                             index-paper-assets,compare-figures,track-run,register-target-artifact,
                             record-comparison,download-paper,run-reproduce} ...

The harness contract, quoted from source

REQUIRED_MATRIX_COLUMNS in paper_replication.py is the exact schema every spec/reproduction_matrix.csv row must have: target_id, kind, paper_locator, source_locator, runner, config, output_path, acceptance_mode, comparison_metric, tolerance, status, report_anchor. Status is one of PLANNED, ACTIVE, MATCHED, BLOCKED, SKIPPED (only one row may be ACTIVE). A row only counts as MATCHED once, per references/harness-contract.md and the validator source: the artifact exists and is not a paper-reference asset; it has wrapper-generated run provenance (track-run, never a hand-placed file); provenance points at a project code/config/spec-trace file, declares method_components and implementation_summary, and is marked baseline_faithful; comparison evidence exists and matches the row's acceptance_mode (numeric-equivalence and distributional-equivalence both reject visual-only metrics such as SSIM; qualitative-structural requires a substantive note); and report/main.tex embeds the target's output_path and report_anchor verbatim.

A minimal two-target workspace, run to the real completion gate

A synthetic two-claim paper was written (a Monte Carlo pi estimate and a structural sparse-identification claim, paper-source/main.tex) so the whole lifecycle could be driven without needing an actual scientific-ML paper. Bootstrap, using the real CLI:

python3 scripts/paper_replication.py bootstrap \
  --project-dir ./case_study --paper-title "A Minimal Two-Claim Demonstration Paper" \
  --paper-slug demo-two-claim --paper-source ../paper-source --main-tex main.tex \
  --author-code-policy forbid_by_default --stack-policy paper-driven --compute-mode local

--paper-source is stored in the manifest and resolved relative to --project-dir, not to the caller's working directory (caught on re-review: ./paper-source from this cwd resolves to the nonexistent case_study/paper-source and inspect-paper fails with Configured source path does not exist, reproduced here; ../paper-source, one level up from case_study/, is the real fix, since paper-source/ and case_study/ are both created directly under skill/codex/paper-replication/). This generates the full scaffold from references/harness-contract.md's required-file list (paper_manifest.json, spec/*.md, spec/reproduction_matrix.csv, todo.md, report/main.tex, plus 18 scripts/*.py workspace utilities) for real, then inspect-paper indexed the one-file TeX tree (tex_file_count: 1).

spec/reproduction_matrix.csv starts as only the header row; the two target rows have to be added by hand before any track-run or register-target-artifact call can reference them, per references/quickstart.md's own step 3. kind here is the report-embedding kind (figure/table, checked by required_report_rows in paper_replication.py), a separate axis from acceptance_mode:

target_id,kind,paper_locator,source_locator,runner,config,output_path,acceptance_mode,comparison_metric,tolerance,status,report_anchor
pi_estimate,table,Section 2 (Claim 1),paper-source/main.tex,code/run_pi_estimate.py,config/pi_estimate.json,artifacts/tables/pi_estimate.json,numeric-equivalence,abs_error,0.01,ACTIVE,pi-estimate-table
sindy_structural,table,Section 3 (Claim 2),paper-source/main.tex,code/run_sindy_structural.py,config/sindy_structural.json,artifacts/tables/sindy_structural.json,qualitative-structural,structural_match,,PLANNED,sindy-structural-table

todo.md's Active target line has to name the same row the matrix marks ACTIVE; validate-progress fails on a mismatch between the two.

Two real reproduction scripts were written under case_study/code/ (case_study/config/ holds their inputs): run_pi_estimate.py (uniform rejection sampling in the unit square, N=200000, numpy) and run_sindy_structural.py (sequential thresholded least squares, library {1, x, x^2, x^3}, over a noisy simulated trajectory of dx/dt = -0.5 x). Both live inside the project directory because track-run's default execution cwd is --project-dir itself, not the caller's shell location. Both were executed through the harness's own run wrapper, not called directly:

python3 scripts/paper_replication.py track-run --project-dir ./case_study \
  --label "pi-estimate baseline" \
  --shell-command "python3 code/run_pi_estimate.py config/pi_estimate.json artifacts/tables/pi_estimate.json" \
  --expected-artifact artifacts/tables/pi_estimate.json

Real result: pi_hat=3.1445, abs_error=0.0029 against the paper's stated < 0.01 bound (N=200000 gives a Monte Carlo standard error of about 0.0029, so this run landed about one standard error from pi, comfortably inside the bound). The structural target's first attempt genuinely failed: at STLSQ threshold 0.05, differentiation noise from np.gradient on the noisy trajectory left three terms above threshold (recovered_nonzero_terms: ["x", "x^2", "x^3"], coefficients x=-0.4862, x^2=-0.0713, x^3=0.0719, structural_match: false) instead of the paper's claimed single term. That run (run-0002) was kept, not deleted, exactly as references/harness-contract.md requires ("Preserve superseded run records rather than overwriting them"); raising the threshold to 0.1 (still a single STLSQ hyperparameter, not a fit to the paper's answer) recovered exactly {x} with coefficient -0.4976 against a true -0.5 (run-0003, structural_match: true).

Both targets then went through the honest, non-adversarial register-target-artifact call (same flags as below, a truthful --implementation-summary, no forbidden markers) against their real successful runs, run-0001 for pi_estimate and run-0003 for sindy_structural, each followed by record-comparison with real metrics (abs_error=0.0029 for pi_estimate; a qualitative-structural note plus coefficient_x=-0.4976 for sindy_structural). register-target-artifact rejects --deviation-notes on a baseline claim ("...: baseline provenance may not carry deviation notes.", reproduced here); a baseline claim has to be paper-faithful with no deviation text, so the correction path is documented in prose and in the kept run-0002 record, not in the provenance field. Each row's status was then flipped to MATCHED in the matrix once its evidence bundle was complete.

Two adversarial cases run against the real, unmodified validators

Forbidden pattern-matching marker, rejected by register-target-artifact:

$ python3 scripts/paper_replication.py register-target-artifact --project-dir ./case_study \
  --target-id pi_estimate --run-id run-0001 --method-label "hard-coded paper constant" \
  --code-path code/run_pi_estimate.py --config-path config/pi_estimate.json \
  --paper-trace-path spec/math_audit.md --seed 20260716 --implementation-kind paper-method \
  --method-component numeric-integration \
  --implementation-summary "this is a pattern generator fit to paper Table 1" --baseline-faithful
{
  "error": "pi_estimate: baseline method evidence contains forbidden pattern-matching markers: pattern generator, fit to paper, hard-coded paper.",
  "ok": false
}

SUSPICIOUS_BASELINE_MARKERS in paper_replication.py is a fixed phrase list ("pattern generator", "fit to paper", "hard-coded paper", and 10 more) scanned across the method label, implementation summary, run command, and code file text. The write is rejected before anything touches disk; the honest provenance record from the real run was never overwritten.

Visual-only metric on a numeric-equivalence target, rejected by validate-progress:

$ python3 scripts/paper_replication.py record-comparison --project-dir ./case_study \
  --target-id pi_estimate --kind table --acceptance-mode numeric-equivalence \
  --note "visual only" --metric ssim=0.99
{"ok": true, ...}   # record-comparison itself does not gate on acceptance mode

$ python3 scripts/paper_replication.py validate-progress --project-dir ./case_study
{
  "details": {"errors": [
    "pi_estimate: numeric-equivalence targets require non-visual metrics beyond SSIM/pixel similarity.",
    "pi_estimate: report/main.tex does not reference artifacts/tables/pi_estimate.json.",
    "pi_estimate: report/main.tex does not include report anchor pi-estimate-table."
  ]},
  "error": "validate-progress failed.", "ok": false
}

This is a real, load-bearing gap to know about operationally: record-comparison writes whatever evidence it is given; the acceptance-mode check that rejects visual-only evidence lives in validate-progress (and only fires once a row's status is MATCHED), not at write time. Re-recording the real abs_error metric and re-running validate-progress cleared that error immediately.

Reaching the real completion gate

With both targets' evidence honest and complete, report/main.tex filled in (target output paths and report_anchor tokens embedded verbatim, per the contract above; \verb|artifacts/tables/pi_estimate.json| rather than an escaped pi\_estimate.json, since the check is a plain substring search over the raw .tex text and an escaped underscore does not match) and compiled with pdflatex to a real report/main.pdf:

$ python3 scripts/paper_replication.py validate-spec --project-dir ./case_study
{"ok": true, "result": {"errors": [], "ok": true}}
$ python3 scripts/paper_replication.py validate-progress --project-dir ./case_study
{"ok": true, "result": {"errors": [], "ok": true}}
$ python3 scripts/paper_replication.py validate-report --project-dir ./case_study
{"ok": true, "result": {"errors": [], "ok": true}}
$ python3 scripts/paper_replication.py validate-completion --project-dir ./case_study
{"ok": true, "result": {"errors": [], "incomplete_targets": [], "matched_target_count": 2,
                          "ok": true, "total_target_count": 2}}

status --project-dir ./case_study (the rehydration command the skill tells an agent to run after every context compaction) reports "completion_ok": true and "next_action": "Completion gate passed. The case study is ready for final review.", reading only repo-local files, exactly as documented. build-paper-pdf was also run against the synthetic paper's own main.tex (compiling the paper being replicated, a separate artifact from the replication report) and produced artifacts/paper_build/main.pdf without incident.

Running the shipped test suite

$ python3 -m pytest scripts/test_paper_replication_unit.py -q
.....F............F......................                                [100%]
2 failed, 39 passed in 2.60s

test_paper_replication_unit.py lives under scripts/, not directly in skill/codex/paper-replication/; running it as python3 -m pytest test_paper_replication_unit.py -q from this directory (an earlier revision of this page showed exactly that) fails with ERROR: file or directory not found: test_paper_replication_unit.py, reproduced here, since the file is one level down. The scripts/ prefix above is the real fix. The 2 failures are test_cluster_delegate_info_prefers_project_cluster_wrapper and test_run_reproduce_uses_project_cluster_wrapper_when_present; both raise PaperReplicationError: compute.mode=cluster but cluster-slurm is not discoverable, because the sibling cluster-slurm skill referenced in references/cluster-integration.md is not installed in this sandbox. That is an environment gap, not a defect in paper-replication itself; installing the cluster-slurm skill alongside this one before relying on compute_mode=cluster is the fix, not a code change.

How to run it in production

• Treat MATCHED as "satisfied its own recorded acceptance rule," not "numerically exact." In the reported case study, of 39 independently re-checked scalar anchor-run observations, 37 fell inside a fixed paper-anchored threshold and 2 did not (a Schrödinger PINN-I run at 4.8e-2 against a 1 percent threshold, and a Navier-Stokes PINN-II run at 16.4 percent against a 10 percent threshold), yet both targets were still recorded MATCHED because the workspace's own acceptance rule for that target accepted them. If you need a stricter external audit, re-check scalar claims against a fixed threshold as a separate pass, the way the paper's own case-study analysis does, rather than trusting MATCHED alone.
• Watch the run-to-run headroom and residual-scale numbers, not just the pass/fail count. The reported headroom model gives posterior-median headroom (log10 units) of 0.51 for PINN-I, 1.75 for PINN-II, and 0.42 for SINDy (roughly 3.2x, 57x, and 2.6x inside threshold on average), but the run-residual scale is largest for PINN-II (posterior median 1.16, about a 14x spread across reruns): completion being stable across runs does not mean fidelity is stable across runs.
• Expect and record correction work; do not treat it as failure. Across the reported corpus, 25 of the tracked executions were later superseded by correction work before final evidence was accepted (21 of those in the two PINN papers). The paper's framing is that this trial-and-correction path is part of the evidence, not overhead to hide.
• Judgment variation is real: two completed workspaces can classify the same claim differently. The reported same-acceptance-rule-type agreement across repeated runs of the same paper ranged from 0.95 (SINDy) down to 0.46 (PINN-II); if you are aggregating MATCHED status across runs or agents, do not assume they used the same kind of evidence for the same claim.

How to maintain it

• Re-run the completion-gate assertions on any change to the evidence-bundle logic. The stdlib-only block above checks target evidence and every workspace-level validator; a regression that bypasses any term reopens the failure mode the gate is designed to close.
• Keep the target-type-specific acceptance rules explicit and versioned. Numeric tolerances (1e-2, 10 percent, 1e-3 in the reported study) are per-paper analysis conventions, not universal constants; do not hard-code a single tolerance across target types.
• Preserve superseded run records rather than overwriting them. The trial-and-correction path (25 superseded executions in the reported corpus) is part of what makes the workspace auditable; a maintenance change that prunes "failed" runs to save space destroys the evidence the completion gate is supposed to produce.
• Re-validate hash-based copied-asset checks after any change to the workspace directory layout. The separation between paper-provided assets and agent-generated outputs is what the hash check relies on; the paper is explicit that this check does not catch every transformed copy, so keep the provenance and comparison checks as the backstop, not the hash check alone.
• Run the shipped test_paper_replication_unit.py suite before upgrading the pinned commit. 39 of 41 tests pass unmodified against e030a7b5d; the 2 that fail in an environment without the sibling cluster-slurm skill installed (test_cluster_delegate_info_prefers_project_cluster_wrapper, test_run_reproduce_uses_project_cluster_wrapper_when_present) are an environment gap, not a code defect, confirmed by reading the raised PaperReplicationError directly. A real regression will fail a different, unrelated test.
• record-comparison does not gate on acceptance_mode at write time; validate-progress does. A visual-only metric recorded against a numeric-equivalence or distributional-equivalence target will write successfully and only get caught the next time validate-progress runs against a MATCHED row. Do not treat a successful record-comparison call as proof the evidence will satisfy the gate; always follow it with validate-progress.

Results

Reported on 12 independent runs (3 per paper) across 4 scientific machine learning papers: physics-informed information field theory (PIFT), two physics-informed neural network papers (PINN-I: forward solutions, PINN-II: coefficient discovery), and sparse identification of nonlinear dynamics (SINDy). Coding agent: Codex with GPT-5.4 at the "Extra High" reasoning setting, author code disallowed.

Paper	Runs	Targets/run	All matched	Elapsed time, median [95% CI] (h)	Superseded executions
PIFT	3	8, 8, 25	yes	2.2 [1.1, 4.4]	3
PINN-I	3	8, 8, 8	yes	5.0 [2.5, 9.9]	11
PINN-II	3	9, 9, 15	yes	6.9 [3.0, 13.4]	10
SINDy	3	20, 20, 20	yes	1.9 [1.0, 4.3]	1

All 12 workspaces reached the completion gate; all 158 recorded targets were MATCHED with report coverage. Of 39 independently re-checked scalar anchor-run observations (13 anchors x 3 runs), 37 fell inside a fixed paper-anchored threshold and 2 did not, while remaining MATCHED under their own workspace's acceptance rule. Same-acceptance-rule-type agreement across repeated runs of a paper ranged from 19/20 (SINDy) to 5/11 (PINN-II).

Failure modes

• Mistaking MATCHED for exact numerical reproduction. It means the recorded, claim-specific acceptance rule was satisfied, not that the reproduced value equals the paper's printed digit for digit; the paper's own PINN-II clean-coefficient errors varied from 7.3 percent to 0.014 percent across reruns, all still under the 10 percent convention used.
• Trusting hash checks alone against copied material. They compare paths and hashes against indexed paper assets and rendered pages, and the authors state plainly that this does not rule out every transformed copy or adversarial reuse; provenance and comparison records are the actual backstop.
• Judging a workflow's effect from this study alone. There is no reported ablation against unstructured prompting, so the numbers characterize what Paper-replication produces, not how much better it is than prompting an agent without the skill.
• Small-corpus overreach. Four papers and three runs each is enough to see within-paper run-to-run variation, but the authors caution the corpus is too small to draw strong between-paper conclusions or to rank paper difficulty from elapsed time or correction-work counts.
• Treating scalar thresholds as ground truth. The numeric tolerances used in the case-study re-analysis (1e-2, 10 percent, 1e-3) are the authors' explicit analysis conventions derived from each paper's reported accuracy scale; a different defensible threshold would shift how many anchors count as "inside," without changing the underlying reproduced values.
• No scalar anchor for non-numeric claims. PIFT contributes zero scalar anchors because its claims (posterior collapse, bimodality, selective identifiability) are distributional and structural; do not force a headroom-style analysis onto a target type it does not fit.

References

• Hans, A., Bilionis, I., Coding-agents can replicate scientific machine learning papers: https://arxiv.org/abs/2607.02134 (code, prompts, and case-study workspaces, pinned at commit e030a7b5dc625acb7cfc1b9b5630161b7a4a1ed2 (2026-07-05), Apache 2.0: https://github.com/PredictiveScienceLab/paper-replication-paper)
• Lopopolo, R., Harness engineering: leveraging Codex in an agent-first world: https://openai.com/index/harness-engineering/
• Starace et al., PaperBench: Evaluating AI's Ability to Replicate AI Research: https://arxiv.org/abs/2504.01848
• Siegel et al., CORE-Bench: Fostering the Credibility of Published Research Through a Computational Reproducibility Agent Benchmark: https://arxiv.org/abs/2409.11363
• Seo et al., PaperCoder / Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning: https://arxiv.org/abs/2504.17192
• Xiang et al., SciReplicate-Bench: Benchmarking LLMs in Agent-Driven Algorithmic Reproduction from Research Papers: https://arxiv.org/abs/2504.00255
• Manheim, D., Garrabrant, S., Categorizing Variants of Goodhart's Law: https://arxiv.org/abs/1803.04585

Related: Evaluating agents · Evaluation integrity and anti-gaming · Workspace-Bench · NatureBench · Loop engineering · Harness architecture · Context and memory · Agentic systems