Copy for agents Markdown

Harness-R1: learning to edit runtime harnesses

Scope: Harness-R1 (arXiv 2608.02276), a method that trains a dedicated 9B harness engineer to convert batches of failed target-agent trajectories into executable runtime hooks. This page covers the failure-packet and patch contracts, four lifecycle intervention points, cold-start SFT plus online GRPO, same-batch reward, released implementation, reported results, and the controls needed to promote a generated patch beyond an evaluation batch. It is the learned-editor case within self-improving harnesses, complements HarnessX's typed foundry and model co-training, and needs the external promotion controls in governing self-modifying agents.

Evidence status, verified 2026-08-05. Claims and numbers are from arXiv v1, submitted 2026-08-03. The implementation was checked at DeepExperience/Harness-R1 commit b9d9853313ae0339f024f3bde860e90519bab53e; its 17 source-only tests and compile checks pass under Python 3. The benchmark experiments were not reproduced because their task assets, model endpoints, and training run require external infrastructure. The NumPy block below was executed and asserted, and the JSON patch template was schema-validated, compiled, and unit-executed against that commit's runtime. scripts/check_release.py and demo/run_demo.py were run. The tool-call probe was run far enough to confirm its argument contract against an unreachable endpoint. Remaining shell snippets are pinned reference templates.

flowchart TB
  BASE["Frozen target agent A<br/>run full task batch B"] --> FAIL["Deterministic extractor<br/>retain and compact failed episodes"]
  FAIL --> ENG["Harness engineer H<br/>cold-start SFT, then online GRPO"]
  ENG --> CAND["Eight candidate JSON patches<br/>per failure packet"]
  CAND --> VALID{"Valid, active, and complete?"}
  VALID -->|"no"| ZERO["Engineer reward = 0"]
  VALID -->|"yes"| PATCH["Install executable overlay P<br/>four lifecycle hook positions"]
  PATCH --> RERUN["Rerun frozen A<br/>on the same full batch B"]
  BASE -.->|"baseline rewards"| DELTA["Mean patched reward<br/>minus mean baseline reward"]
  RERUN --> DELTA
  DELTA --> GRPO["Group-relative advantage<br/>update engineer only"]
  ZERO --> GRPO
  GRPO --> ENG

What it is

Harness-R1 separates the agent being improved from the model that writes the improvement. A frozen target agent runs a batch of interactive tasks. A deterministic extractor retains only failed episodes and compacts their constraints, selected action-observation excerpts, outcomes, and necessary environment state into one failure packet. A separate harness engineer reads that packet once and emits one reusable JSON patch containing Python code hooks. The engineer neither answers tasks nor participates in the patched rollout.¹

The patch wraps the target at episode initialization, before a decision, before an action reaches the environment, and after environment feedback. The target model weights stay fixed within a training stage. Every valid candidate is installed independently and the target reruns the complete original batch, including tasks that succeeded before the edit. Only the engineer receives a parameter update from the resulting reward.¹

This is not an online self-edit inside one episode. The paper's formulation has no iterative patch refinement within an instance and no persistent patch memory across batches. Its co-evolution result is also a two-stage experiment, not a continuing alternating loop: the authors first fine-tune the target agent, then train a target-specific engineer for that stronger frozen target.⁸

Why use it

Syntactic validity and a plausible diagnosis do not establish that a runtime edit helps. Harness-R1 makes the target's realized task outcome the editing signal. A valid patch that regresses the batch receives a negative reward; an invalid, inert, or ultimately incomplete patch receives zero. This puts action mediation, context guidance, state tracking, and recovery under one outcome-grounded objective rather than training the editor to imitate teacher text alone.²

The paper reports the following single-episode results on a frozen Qwen3.5-9B target. Avg. is the equal-weight mean of ALFWorld success, WebShop success, and DBBench success. Reflection is excluded here because its reported success is cumulative over two episodes and is not comparable to these success@1 rows.³

Method	ALFWorld	WebShop	DBBench	Avg.
Default harness	40.6	31.2	61.0	44.3
Supervised-only engineer	39.4	38.6	61.3	46.4
Harness-R1	53.2	42.2	65.3	53.6
Target-agent SFT	71.2	42.6	63.7	59.2
Target-agent SFT + target-specific Harness-R1	84.0	43.0	65.7	64.2

The outcome-trained editor raises the vanilla target from 44.3% to 53.6%, 9.3 percentage points, and finishes 7.1 points above the supervised-only engineer. The strongest fixed frontier editor in the paper, GLM-5.2, reaches 48.8%. After target-agent SFT, a newly trained target-specific engineer raises the displayed average from 59.2% to 64.2%, another 5.0 points.³

Two experiments test whether the editor merely memorizes its training target or evidence batch. Applied without per-target retuning, the learned editing policy produces fresh target-specific patches for 20 unseen target models and reports a mean 7.06-point gain; every target-level average is positive, although 3 of 60 unseen-target benchmark cells regress. Given only 10 failures per benchmark and seed, Harness-R1 reports an 8.9 +/- 1.5 point improvement over 1,270 held-out tasks across three seeds, while two fixed frontier editors average negative under the matched protocol.⁴

When to use it (and when not)

• Use it for interactive agents with executable, localized failure modes. Tool misuse, premature actions, state loss, protocol violations, repeated actions, and failed recovery map directly to the four hook positions.
• Require a reproducible task-level reward. WebShop supplies shaped reward; ALFWorld and DBBench supply binary success. An LLM judgment of patch plausibility is not the method's signal.
• Use it where baseline and patched executions can be paired exactly. Task identity, environment state, target checkpoint, decoding configuration, and seeds must match. Otherwise the delta measures drift.
• Prefer it when the target model cannot be retrained for every repair. Harness editing changes the runtime around a frozen model and remains complementary to later target-agent training.
• Do not use the paper's same-batch reward as a production promotion rule. It is a transductive training objective. It can improve its evidence batch while regressing unseen tasks, and a positive mean can conceal regressions on individual previously successful tasks.
• Do not use the released hook runner as the only hostile-code boundary. It validates an AST and restricts builtins, time, and executed lines, but still executes generated Python in process. Put untrusted patches behind a process, container, or stronger sandbox boundary (agent sandboxing).
• Budget for evaluation-dominated training. Each failure packet samples eight patches, and each valid patch reruns the frozen target on the full task batch. The reference SFT, engineer RL, and target SFT stages each used one node with eight NVIDIA H800 GPUs.²

Architecture

Four lifecycle hooks

The patch schema permits only add_code_hook actions in the strict training protocol. Each action defines exactly one hook(ctx, nb) function. ctx is a benchmark-specific read-oriented runtime snapshot; nb is mutable per-episode notebook state. The host accepts only structured return effects.⁶

Hook	Invocation	Permitted role
on_init	Before the first target decision	Initialize notebook state; add reusable skills or a tool hint
make_pre_hint	Before a target decision	Inject state-conditioned guidance without executing an action
on_before_action	After target proposal, before environment execution	Block and reprompt; where supported, rewrite or force the pending action
on_post_step	After environment feedback	Update state, inject recovery guidance, or where supported schedule an action

The action surface is benchmark-specific. DBBench permits soft guidance and blocking but does not execute SQL rewrites or forced commits. ALFWorld exact mediated actions must come from the current admissible set. WebShop mediation is intended for narrow observable mistakes such as buying over budget, skipping a required option, or repeating an action. Task indices, answers, product IDs, and numbered ALFWorld instances are forbidden.⁵

Failure packet and reward

For task batch B of size n, baseline reward R_i^0, and patched reward R_i^P, the engineer reward is:

$$ \Delta_B(P) = \frac{1}{n}\sum_{i=1}^{n}(R_i^P - R_i^0), \qquad r(B,P) = \begin{cases} \Delta_B(P) & \text{valid and complete} \ 0 & \text{otherwise.} \end{cases} $$

The failure packet contains only failed baseline trajectories, but the reward covers all tasks in the batch. Rerunning the successful tasks is the only regression pressure in the objective. It is aggregate pressure, not a no-regression invariant: one new success and one lost success cancel.

Cold start and online GRPO

Cold-start SFT teaches the output protocol and a prior over executable edits. A teacher generates candidates on task batches disjoint from the RL batches; the pipeline retains at most one executable, complete, non-regressive response per packet. The paper says roughly 1,000 examples, while Appendix B gives the exact retained count as 877: 381 WebShop, 248 ALFWorld, and 248 DBBench.²

Online GRPO then samples K=8 patches from the current engineer for each packet. Rewards are centered and divided by their within-packet standard deviation, and the resulting sequence-level advantage is shared across response tokens in a clipped objective. The reference configuration uses lower and upper clipping of 0.20 and 0.28, truncated importance weights capped at 2.0, temperature 0.7, no entropy bonus, and no explicit KL loss. The released training script still passes the KL flags with a coefficient of 0.00, so the term is present but contributes nothing. The released Relax snapshot uses PyTorch's sample standard deviation plus 1e-6 in the denominator, and group standard-deviation normalization is on unless explicitly disabled.²

Patch validation and execution

At the pinned release, a code hook is limited to 8,000 source characters and 1,200 AST nodes. The compiler requires exactly one top-level hook(ctx, nb), permits at most five top-level helpers, and rejects imports, classes, lambdas, async syntax, while, filesystem access, dynamic evaluation, dunder access, and other escape surfaces. Runtime execution defaults to 50 ms and 2,000 traced lines; an exception, timeout, invalid return shape, or unsupported effect degrades to no effect.⁶

These checks prevent common accidental and generated-code failures. They do not provide kernel-enforced isolation, and degrading to no effect makes hook-failure telemetry part of correctness rather than optional observability.

How to use it

The public release includes an offline patch demo and source-only checks. Every command in this pinned sequence was executed; the full benchmark commands were not run:

git clone https://github.com/DeepExperience/Harness-R1.git
cd Harness-R1
git checkout b9d9853313ae0339f024f3bde860e90519bab53e
python3 scripts/check_release.py
python3 demo/run_demo.py --no-color

The demo is a stored WebShop evaluation shipped with the release, not the paper's case study. It reports its own batch moving from 1/10 to 5/10; the WebShop example in Appendix H moves from 2/10 to 5/10. Do not read one as a reproduction of the other.

Full evaluation requires benchmark assets, separate engineer and frozen-target endpoints, and environment-specific interpreters. Copy configs/eval/endpoints.env.example, pin every endpoint model and chat-template option, then run one of scripts/eval_webshop.sh, scripts/eval_alfworld.sh, or scripts/eval_dbbench.sh. The target endpoint must return structured message.tool_calls; XML-like tool text in message.content is not equivalent and produces zero reward in this release.⁶

Because that failure is silent, probe the target endpoint before committing to a long run rather than discovering flat zero rewards afterward. Use $ROLLOUT_TOOL_CHOICE from the endpoint file so the probe matches the tool-choice mode the rollout will request:

python3 scripts/probe_openai_tool_calls.py \
  --base-url "$TARGET_BASE_URL" --model "$TARGET_MODEL" \
  --tool-choice "$ROLLOUT_TOOL_CHOICE" \
  --output outputs/tool_call_probe.json

The probe writes a JSON report and exits non-zero when any attempt fails to return structured tool calls. Note that --tool-choice and --output are required arguments; the release README prints this command without them, so the README form exits 2 on argument parsing before it ever contacts the endpoint.⁷

Cross-target evaluation must generate patches from each target's own failures, then apply those patches to that same target. The release exposes this as a generate-only run followed by a rerun with PATCH_SOURCE_ROOT. Reusing one fixed patch would test patch transfer, not the paper's editor-policy transfer claim.

How to develop with it

Start with the narrow patch language, not arbitrary repository edits. The canonical shape below is a shortened reference template based on the released WebShop example. It was schema-validated, compiled, and unit-executed against the pinned runtime; it was not run through a WebShop benchmark rollout and still needs benchmark-specific tests before use:

{
  "schema_version": "harness-r1-patch-v1",
  "benchmark": "webshop",
  "description": "Block a purchase until every required option is selected.",
  "actions": [{
    "type": "add_code_hook",
    "hook": "on_before_action",
    "code": "def hook(ctx, nb):\n    action = ctx.get('action', {})\n    predicates = ctx.get('predicates', {})\n    if action.get('value_normalized') == 'buy now' and predicates.get('required_options_unselected'):\n        return {'kind': 'block_and_prompt', 'message': 'Select every required visible option before buying.'}\n    return None"
  }]
}

Every patch needs three test layers:

1. Protocol tests reject malformed JSON, unsupported hooks or effects, repeated hooks, task-specific identifiers, imports, and runtime no-ops.
2. Hook unit tests drive normal, boundary, and hostile contexts through the compiled hook and assert both the structured effect and notebook state.
3. Paired rollout tests rerun the full batch and a separate held-out regression split under identical target and environment fingerprints. Store per-task deltas, not only the mean.

The following NumPy block reproduces the displayed benchmark average, implements the same-batch delta and released group normalization, and attacks three assumptions: task identity can drift, aggregate reward can cancel a regression, and zero-reward invalid outputs can receive positive relative advantage when the valid candidates are harmful. It also constructs a patch that improves its training batch and regresses a held-out batch.

# harness_r1_objective.py - executed: same-batch delta, GRPO group
# normalization, identity and completion guards, cancellation, and transductive
# overfit. This validates the algorithmic shape, not the paper's model results.
import numpy as np


def patch_reward(base_ids, base_reward, patched_ids=None, patched_reward=None,
                 *, valid=True, complete=True):
    """Equation (1), with release-style zero reward for unusable patches."""
    if not valid or not complete:
        return 0.0
    base_ids = np.asarray(base_ids)
    patched_ids = np.asarray(patched_ids)
    base_reward = np.asarray(base_reward, dtype=float)
    patched_reward = np.asarray(patched_reward, dtype=float)
    assert np.array_equal(base_ids, patched_ids), "task identity drift"
    assert base_reward.shape == patched_reward.shape == base_ids.shape
    assert np.isfinite(base_reward).all() and np.isfinite(patched_reward).all()
    return float(np.mean(patched_reward - base_reward))


def group_advantage(rewards):
    """Released Relax shape: within-group mean, sample std, epsilon 1e-6."""
    rewards = np.asarray(rewards, dtype=float)
    assert rewards.ndim == 1 and rewards.size > 1
    centered = rewards - rewards.mean()
    return centered / (centered.std(ddof=1) + 1e-6)


# Reproduce the paper's displayed equal-benchmark averages for the vanilla target.
baseline_metrics = np.array([40.6, 31.2, 61.0])  # ALFWorld, WebShop, DBBench
harness_metrics = np.array([53.2, 42.2, 65.3])
baseline_avg, harness_avg = baseline_metrics.mean(), harness_metrics.mean()
assert round(baseline_avg, 1) == 44.3
assert round(harness_avg, 1) == 53.6
assert round(harness_avg - baseline_avg, 1) == 9.3

ids = np.arange(4)
train_gain = patch_reward(ids, [1, 0, 0, 0], ids, [1, 1, 0, 0])
heldout_loss = patch_reward(ids, [1, 1, 1, 0], ids, [1, 0, 1, 0])
cancelled = patch_reward(ids, [1, 1, 0, 0], ids, [1, 0, 1, 0])
assert train_gain == 0.25 and heldout_loss == -0.25
assert cancelled == 0.0  # one rescue and one regression disappear in the mean

# Invalid/no-op outputs score zero. If all valid edits are harmful, group-relative
# normalization makes zero better than the group mean and therefore positive.
candidate_rewards = np.array([-0.4, -0.3, -0.2, -0.1, 0.0, 0.0, 0.0, 0.0])
adv = group_advantage(candidate_rewards)
assert adv[0] < 0.0 and np.all(adv[4:] > 0.0)
assert patch_reward(ids, [0, 0, 0, 0], valid=False) == 0.0
assert patch_reward(ids, [0, 0, 0, 0], complete=False) == 0.0
assert np.array_equal(group_advantage(np.zeros(8)), np.zeros(8))

try:
    patch_reward(ids, [0, 0, 0, 0], ids[::-1], [0, 0, 0, 0])
    raise AssertionError("identity mismatch must fail")
except AssertionError as exc:
    assert str(exc) == "task identity drift"

print(f"table average: {baseline_avg:.1f} -> {harness_avg:.1f} "
      f"({harness_avg - baseline_avg:+.1f} pp)")
print(f"same-batch: train={train_gain:+.2f} held-out={heldout_loss:+.2f} "
      f"cancelled={cancelled:+.2f}")
print(f"group advantage: harmful={adv[0]:+.3f} invalid/no-op={adv[-1]:+.3f}")
print("all objective assertions passed")

Executed output:

table average: 44.3 -> 53.6 (+9.3 pp)
same-batch: train=+0.25 held-out=-0.25 cancelled=+0.00
group advantage: harmful=-1.739 invalid/no-op=+0.791
all objective assertions passed

The positive invalid/no-op advantage is an implementation consequence, not a reported paper result. Cold-start SFT, strict validation, and executable-example filtering reduce its frequency, but the release sets the validity bonus to 0.0. Track invalid and runtime-no-op rates per update; if either rises while task reward stalls, the editor may be learning abstention through malformed output rather than useful patches.

How to maintain it

• Fingerprint both sides of every comparison. Record target checkpoint, model-serving revision, chat template, tool schema, target temperature, benchmark data hash, task manifest, seed, base harness commit, patch schema, and engineer checkpoint.
• Pin the released artifacts. The source checks above use commit b9d9853; the model repository resolved to revision c1ae0389d53bfac326eec21285b16a26e121ff95 on 2026-08-05. A moving main invalidates reproduction claims.
• Rebuild baseline caches after any target change. Cached failure packets and rewards belong to one frozen target and base runtime. Do not compare a new target against old baselines.
• Separate invalid output, target regression, and infrastructure failure. Invalid or inactive patches intentionally score zero; valid regressions score negative; endpoint, database, or worker failures are missing evaluations. Retrying infrastructure failures until they turn positive biases the run.
• Keep per-task ledgers. Store which hooks fired, their structured effects, baseline and patched outcomes, and previously successful tasks lost. Mean delta alone cannot support rollback or diagnose cancellation.
• Re-audit the executor when Python changes. AST allowlists, safe builtins, line tracing, signal timeouts, and effect normalization are version-sensitive security code.
• Retrain or at least revalidate after target-agent updates. The paper trains a separate engineer for its SFT target. The vanilla-target engineer is not claimed to remain optimal after the actor changes.

How to run it in production

The paper evaluates one generated patch on controlled benchmark reruns; it does not present a live promotion system. A production design needs an outer state machine that the engineer cannot edit:

1. Generate a candidate from a bounded, redacted failure packet.
2. Compile it in a locked build environment and execute it in an isolated worker with no production credentials.
3. Replay the evidence batch, then a disjoint regression set. Require task-level floors in addition to a positive mean.
4. Shadow on current traffic and measure hook firing rate, task success, tool-policy violations, latency, token use, and executor failures.
5. Sign the patch artifact and its evaluation manifest; canary it behind per-hook kill switches.
6. Promote only through an external policy gate, retain the previous artifact, and roll back automatically on a floor breach.

Add a patch registry if interventions must persist across batches. Add a scheduler if several patches overlap, because the paper does not define precedence, conflict resolution, or composition across independently generated overlays. Add cost to the promotion objective: the authors explicitly leave inference-efficiency reward for future work, so task gain can otherwise purchase arbitrary context, latency, or tool-call overhead.⁸

Failure modes

• Evidence-batch overfitting. The same tasks produce failures and reward. Held-out evidence is reported separately in the paper but is not part of the training objective.
• Aggregate cancellation. A rescued failure and a regressed success can produce zero delta. Promotion needs per-task regression floors.
• Invalid-output advantage. Zero-reward invalid or inert patches can sit above a negative group mean and receive positive GRPO advantage.
• Broad action override. A plausible stage estimate can force the wrong action. In the paper's ALFWorld case study, a Gemini-3.5-Flash patch reduces full-set success from 41.6% to 35.4%; its worst batch falls from 7/10 to 0/10.⁹
• Notebook state drift. Incorrect state updates make later hints and guards confidently wrong, especially on multi-object tasks. The paper's own ALFWorld patch rescues six failures but regresses one prior success and still loops on one two-object task.⁹
• Task-identity mismatch. Integer indices alone do not prove a paired WebShop comparison. The release pins WebShop to goal seed 233 by default and rejects a run whose task manifest disagrees with the expected seed, so the guarantee is manifest agreement rather than one fixed seed value.
• Silent hook failure. Exceptions, timeouts, and malformed effects degrade to no effect. Without hook-outcome telemetry, the patch appears installed while doing nothing.
• Target or substrate drift. A patch and engineer are learned against one target-runtime pair. Changing weights, tool schemas, prompts, or environment state breaks causal attribution.
• Protocol-mismatched comparison. Reflection's two-episode cumulative success cannot be ranked against the paper's single-episode methods.
• Missing cost signal. The reward contains task utility only, so runtime overhead is unpriced.
• Sandbox overclaim. AST restriction is useful defense in depth, not equivalent to an OS security boundary for generated code.

References

• Shao et al., Harness-R1: Learning to Edit Executable Runtime Harnesses from Agent Failure Trajectories (arXiv 2608.02276v1): https://arxiv.org/abs/2608.02276
• Harness-R1 source release, pinned commit b9d9853313ae0339f024f3bde860e90519bab53e: https://github.com/DeepExperience/Harness-R1/tree/b9d9853313ae0339f024f3bde860e90519bab53e
• Harness-R1 model release, pinned revision c1ae0389d53bfac326eec21285b16a26e121ff95: https://huggingface.co/ShaoShuai0605/Harness-R1/tree/c1ae0389d53bfac326eec21285b16a26e121ff95
• Shao et al., DeepSeekMath (GRPO): https://arxiv.org/abs/2402.03300
• Yao et al., WebShop: https://arxiv.org/abs/2207.01206
• Shridhar et al., ALFWorld: https://arxiv.org/abs/2010.03768
• Liu et al., AgentBench (DBBench): https://arxiv.org/abs/2308.03688

Related: Self-improving harnesses · HarnessX · Frontis-MA1 and OpenMLE · Harness architecture · Governing self-modifying agents · Agent sandboxing and isolation · Evaluating agents · Evaluation integrity and anti-gaming · Agent observability · GRPO · RL libraries for LLMs

---

1. arXiv 2608.02276v1, Sections 3.1 and 3.2. The frozen target runs batch B; a deterministic extractor compacts failed episodes into s_B; the engineer reads it once and emits one batch-conditioned overlay; the same frozen target reruns all tasks; only the engineer is updated. No within-instance refinement or persistent patch memory is used. ↩↩

2. arXiv 2608.02276v1, Equations 1 to 4, Algorithm 1, and Appendices B and D. Exact cold-start count 877; roughly 1,500 RL failure packets; K=8; rollout batch 4 prompts, global batch 32 sequences, four rollout iterations per update; 28,672-token prompt and 12,288-token response limits; lower/upper clip 0.20/0.28; truncated importance weight cap 2.0; no entropy or explicit KL coefficient; all three training stages on one 8x H800 node. Release behavior cross-checked against configs/rl/mixed_codepatch.yaml, scripts/train_engineer_rl.sh, and relax/utils/utils.py at commit b9d9853. ↩↩↩↩

3. arXiv 2608.02276v1, Table 1 and Section 4.2. The 44.3 to 53.6 and 59.2 to 64.2 deltas use the displayed equal-weight Avg. cells. Recomputing those averages from the displayed one-decimal component cells gives 59.17 and 64.23, which round to the printed 59.2 and 64.2. The rounded cells differ by exactly 5.0 while the unrounded averages differ by 5.07, so the paper's stated +5.0 points is a difference of rounded values; unreported higher-precision component scores account for the 0.07. ↩↩

4. arXiv 2608.02276v1, Sections 4.3 and 4.4, Tables E.1 and F.1. Twenty target agents are unseen during editor training; each supplies its own failure traces and receives a newly generated patch. Across the full 21-target matrix the paper reports 56 improvements, 4 unchanged cells, and 3 regressions. Appendix E names each regression: WebShop on Llama-3.1-70B (-0.4), ALFWorld on Qwen2.5-72B (-2.0), and WebShop on Gemma-4-31B-it (-0.2). None of the three is the primary Qwen3.5-9B target that Table E.1 marks with a dagger, so all three fall inside the 60 unseen-target cells. The four unchanged cells are located only as "all on WebShop" and are not attributed to named targets. Held-out protocol: 10 failures per benchmark and seed, 1,270 other tasks pooled, three matched seeds. ↩

5. arXiv 2608.02276v1, Appendix A.2 and Table C.1. The benchmark-specific context and return contracts constrain each hook; the host runtime, not generated code, interprets structured effects. ↩

6. Verified against DeepExperience/Harness-R1 commit b9d9853 on 2026-08-05. harness_r1_patch.py defines and normalizes the patch schema; the strict path calls require_code_hook_only_patch; code_runner.py implements the AST restrictions, safe builtins, 8,000-character and 1,200-node limits, 50 ms and 2,000-line runtime defaults, and no-effect failure behavior. python3 scripts/check_release.py passed 17 unit tests plus shell syntax and Python compile checks. ↩↩↩

7. Verified at commit b9d9853 on 2026-08-05. scripts/probe_openai_tool_calls.py declares --base-url, --model, --tool-choice, and --output as required. Running the README's two-flag form exits 2 with error: the following arguments are required: --tool-choice, --output; adding both parses correctly, performs its three attempts, and exits 1 against an unreachable endpoint. ↩

8. arXiv 2608.02276v1, Sections 3.1 and 5. The reward is same-batch and transductive; there is no iterative refinement or persistent patch memory across batches. Multi-round alternating co-evolution, held-out reward, and inference-efficiency terms are stated as future work. ↩↩

9. arXiv 2608.02276v1, Appendix H. The reported Gemini regression is a valid executable patch, not a parse failure. The selected Harness-R1 ALFWorld patch changes its ten-task batch from 1/10 to 6/10 by rescuing six failures and regressing one success. ↩↩