Skip to content
Markdown

autoresearch-rl

Scope: autoresearch-rl as an outer controller for reinforcement-learning and other model-training experiments. The page separates the public 0.4.0 package from the later supplied source snapshot, explains where RL actually occurs, builds a complete local campaign with a separate evaluator, and covers recipe development, framework extension, and production operation. It complements GRPO, PPO, RLVR, autonomous experimentation loops, and the post-training system map.

Verification boundary. The public wheel autoresearch-rl==0.4.0, uploaded on 2026-05-30, was installed in a clean virtual environment. Its CLI, sdist tests, and local campaigns were executed; the sdist failures were traced to example and recipe fixtures absent from both published artifacts. A later private checkout dated 2026-07-30 was audited and exercised for the source-only surface, but that repository is not anonymously resolvable, so readers cannot reproduce those parts from a cited public source. No paid GPU campaign was launched. GPU commands below are source-derived reference templates, not newly reproduced cloud results.

What it is

autoresearch-rl is an experiment controller. It chooses a candidate configuration or code change, runs a training target, reads one objective metric, keeps a new best candidate, records the result, and repeats until a guard stops the campaign. It can wrap SFT, DPO, GRPO, a classifier, or a non-ML program. It is not itself a distributed trainer or rollout engine.

The name hides three different learning loops. Keeping them separate prevents category errors.

Layer Policy being improved State and action Reward or gate
Inner model training Language model or task policy Prompts, completions, tokens, tool actions Verifier, reward model, preference, or task return
Bounded experiment search Experiment proposer Trial history to hyperparameter tuple or code diff +1 for a new best in the learned proposer; the configured scalar decides keep/discard
Source-only online loop Served model across rounds Scored production experience to candidate adapter and weight update Held-out gain, canary tolerance, alarms, optional human approval

An inner GRPO trial may therefore sit inside an outer grid, LLM, or PPO search. The inner GRPO policy learns which completions to produce. The outer policy learns or decides which learning rate, group size, or implementation to try next.

Public release versus supplied source

The package version is not a sufficient capability identifier. The supplied checkout still declares version 0.4.0 even though it contains months of later work.

Surface Public PyPI 0.4.0 Later private snapshot
Bounded search run, run-one, resume, status, validate, print-config, upload Present, with later fixes
Targets command, HTTP, Basilica Same, plus optional persistent rollout and environment roles on Basilica
Policies static, grid, random, LLM parameters, LLM ensemble, LLM diff, hybrid, learned PPO Same eight
Continuous learning Not in the installed CLI continual, serve, online, observe, slo, and campaign
Extras http, basilica, chart, s3, gcs, cloud, dev Adds disagg and online
Scaffolding init command exists, but the wheel contains no bundled examples Bundled minimal example exists; copied file paths are partly rewritten
Version query CLI and importlib.metadata report 0.4.0; autoresearch_rl.__version__ reports 0.2.0 Distribution and module both report 0.4.0

All eight policy names, including llm_ensemble, are already in 0.4.0's PolicyConfig.type literal and dispatcher. What the public release does not contain is the continuous-learning surface, the two extras that support it, and the bundled example payload. The PyPI metadata links to a GitHub repository, but anonymous access returns HTTP 404. Public users can install 0.4.0; they cannot assume that source-only commands or examples are available, and they cannot inspect the source at all.

The file and process contract

A defensible recipe uses at least three executable files, even though the project describes a two-script model.

File Responsibility Mutation rule
prepare.py Materialise train and evaluation inputs, schemas, task pools, and fixed constants Frozen
train.py or trainer.py Train a candidate and write a model or adapter artifact The only LLM-mutable file
eval.py Load the candidate, score held-out data, and print the objective Frozen and outside diff scope
program.md Tell the diff proposer what may change, what must remain, and how success is measured Frozen prompt input
config.yaml Bind objective, target, policy, budgets, paths, and telemetry Operator-owned

prepare.py alone does not enforce evaluation integrity. In many shipped recipes, mutable training code computes and prints the objective. A diff proposer can then change the score it is trying to maximise. A separate eval_cmd closes that direct path, but not the whole security problem: all local subprocesses normally run as the same user and can read the same files. Secret held-out data or hostile generated code requires an evaluator in another security domain, such as a service with a narrow candidate-artifact API.

Preparation cadence also depends on the target:

  • CommandTarget runs prepare_cmd once per controller process. A new process or resume may prepare again.
  • BasilicaTarget chains preparation into each newly deployed trial container, so it runs per trial.
  • HttpTarget does not implement preparation. The remote service owns that lifecycle.

Parameters arrive as JSON in AR_PARAMS_JSON and individually as AR_PARAM_<UPPERCASE_NAME>, with the value passed through str(). Every trial also receives AR_RUN_DIR, AR_PROGRESS_FILE, and AR_CONTROL_FILE, the last two absolutised so the subprocess working directory cannot change where they resolve. Conditionally it receives AR_MODEL_DIR (when telemetry.model_output_dir is set), AR_DATASET_PATH (when --dataset is given, after the controller resolves hf://, s3://, gs://, or https:// with its own credentials), AR_MODEL_NAME (when --model is given), and AR_SEED with PYTHONHASHSEED (when controller.seed is set). Nothing else is injected. AR_ITER is read by emit_progress to stamp each report but is never set by the command target, so every progress line in a local campaign carries "iter":0 regardless of the real iteration; use AR_RUN_DIR to tell iterations apart. A parameter name beginning with AR_ is rejected by validate as reserved_param_key.

Why use it

  • One outer loop for different trainers. A command, HTTP service, or Basilica deployment can expose the same parameter and metric contract. The inner trial may use TRL, verl, custom PyTorch, or a standard executable.
  • A durable experiment ledger. Every iteration writes status, decision, metric, parameters, comparability metadata, manifests, and traces. New best candidates receive version records.
  • Code search has a narrow mutation surface, once you configure it. Diff policies receive one mutable source file, and contract checks reject diffs naming the frozen or program file, out-of-scope basenames, empty patches, and removal of configured required calls. The scope check is built only when frozen_file and program_file are both set, while the schema requires just mutable_file, so the minimum viable llm_diff config has no scope check at all. Set all three.
  • Budgets and failure states are first-class. Wall time, iteration count, no-improvement streak, rolling failure rate, checkpointing, and serial Basilica GPU-hour projections bound a campaign.
  • Local-to-cloud progression is direct. The command target catches file, metric, and subprocess defects before a paid target is selected.
  • The source snapshot adds a controlled online path. Shadow mode, a file-backed human approval interlock, canary and held-out gates, trust-region checks, alarms, safe-stop commands, and SLO rendering support staged adoption.

These controls improve auditability. They do not turn generated Python into trusted code, prove that a reward is valid, or make a local held-out set secret.

When to use it (and when not)

Use it when all of the following hold:

  • One trial can be launched behind a command, one HTTP request, or one Basilica job.
  • Candidate quality reduces to one scalar whose direction is explicit.
  • Training writes a candidate artifact that a fixed evaluator can score.
  • The search space is small enough for a baseline and a declared budget.
  • Every winning trial will be replayed independently before a production promotion.

Choose the proposer by evidence, not novelty.

Policy Selection behaviour Appropriate use Important limit
static Empty parameter dictionary One fixed recipe, smoke tests No search
grid Cartesian product in key order Small baseline spaces Cycles after exhaustion; campaign forecasting may stop at five results
random Seeded uniform choice per parameter Larger baseline spaces Repeats are possible
llm One provider proposes allowed parameter values from history Compare against grid/random on an equal budget Falls back to random on provider or parse failure, after up to 240 s of retry backoff
llm_ensemble Multi-provider parameter proposal Provider diversity experiments Requires a non-empty providers list; same silent random fallback
llm_diff Unified diff against the mutable file Algorithm or implementation search Serial; generated code is not safely sandboxed
hybrid LLM parameters, then diffs after a stall Small param phase followed by code search Requires provider credentials; stays serial
learned Numpy PPO over discrete parameter combinations Small research probes of outer-loop learning Deterministic argmax and finite-difference updates make it unsuitable for a large action space

Do not use it as a replacement for FSDP, Megatron, a rollout scheduler, or a trainer. Put those systems inside the trial. Do not use local diff mode for untrusted code with network credentials or readable secrets. Do not reduce a multi-objective release decision to a scalar without retaining hard constraints for safety, regression, latency, and cost.

Prefer a conventional scheduler or hyperparameter service when the need is only large-scale Bayesian optimisation, multi-fidelity search, cluster packing, or mature pruning. Prefer an ordinary CI pipeline when every candidate is a deterministic code change and no model-guided proposer is needed.

Architecture

flowchart TB
  SPEC["Fixed objective, evaluator, budget, and recipe contract"] --> PROP["Outer proposer: grid / random / LLM / diff / PPO"]
  PROP --> CAND["Candidate parameters or train.py diff"]
  subgraph TRIAL["One trial"]
    PREP["prepare.py: materialise inputs"] --> TRAIN["Mutable trainer: SFT / DPO / GRPO / other"]
    TRAIN --> ART["Candidate model or adapter in AR_RUN_DIR / AR_MODEL_DIR"]
    ART --> EVAL["Frozen eval.py or external evaluator"]
  end
  CAND --> TRAIN
  EVAL -->|"one scalar + status"| DEC{"better than incumbent?"}
  DEC -->|"yes"| KEEP["Version record and candidate artifacts"]
  DEC -->|"no"| DROP["Discard decision, still logged"]
  KEEP --> LEDGER["Ledger, manifests, trace, checkpoint"]
  DROP --> LEDGER
  LEDGER --> GUARD{"budget / failures / plateau / forecast"}
  GUARD -->|"continue"| PROP
  GUARD -->|"stop"| RESULT["Bounded result"]
  KEEP -.->|"source-only online gate"| CANARY["Held-out + canary + alarms + optional approval"]
  CANARY -.->|"promote"| SERVE["Persistent serving fleet hot-swap"]

The outer controller normalises a maximisation objective by negating it internally, so lower internal score is always better. The ledger preserves the original metric value. A candidate is kept only when it strictly improves the best score.

How to use it

1. Install the surface that exists

For the public bounded loop, pin the release and use distribution metadata for version checks:

python3 -m venv .venv
./.venv/bin/pip install 'autoresearch-rl==0.4.0'
./.venv/bin/autoresearch-rl --version
./.venv/bin/python -c 'from importlib.metadata import version; print(version("autoresearch-rl"))'

Do not rely on autoresearch_rl.__version__ in public 0.4.0. Do not start with autoresearch-rl init: the public wheel and sdist omit the example payload, so init lists no available examples.

If you hold a source checkout, create an environment from it and pin the exact commit in the experiment record, because the declared package version does not identify the capabilities:

git rev-parse HEAD                       # record this in the run manifest
uv sync --extra dev
uv run autoresearch-rl --help

Continuous source-only work also needs target extras that the public release does not define:

uv sync --extra basilica --extra disagg --extra online

2. Build a local campaign with a real evaluator boundary

The following CPU example fits y = 2x + 1. It was executed against public 0.4.0. It has separate train and evaluation data and stores each candidate in its own AR_RUN_DIR. The separation makes result flow explicit, but it is a functional test rather than a hostile-code sandbox because the mutable process can still read data/eval.jsonl.

prepare.py is frozen:

from __future__ import annotations

import json
from pathlib import Path


def write_jsonl(path: Path, rows: list[dict[str, float]]) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    text = "".join(json.dumps(row) + "\n" for row in rows)
    path.write_text(text, encoding="utf-8")


write_jsonl(Path("data/train.jsonl"), [
    {"x": -2.0, "y": -3.0},
    {"x": -1.0, "y": -1.0},
    {"x": 0.0, "y": 1.0},
    {"x": 1.0, "y": 3.0},
])
write_jsonl(Path("data/eval.jsonl"), [
    {"x": 2.0, "y": 5.0},
    {"x": 3.0, "y": 7.0},
])
print("prepared_rows=6")

train.py is mutable. It emits a training signal and writes a candidate, but does not print the release objective:

from __future__ import annotations

import json
import os
from pathlib import Path

from autoresearch_rl.target.progress import emit_progress


def read_jsonl(path: Path) -> list[dict[str, float]]:
    return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]


params = json.loads(os.environ.get("AR_PARAMS_JSON", "{}"))
learning_rate = float(params.get("learning_rate", 0.02))
steps = int(params.get("steps", 40))
rows = read_jsonl(Path("data/train.jsonl"))
w = b = 0.0

for step in range(1, steps + 1):
    residuals = [w * row["x"] + b - row["y"] for row in rows]
    grad_w = 2.0 * sum(r * row["x"] for r, row in zip(residuals, rows)) / len(rows)
    grad_b = 2.0 * sum(residuals) / len(rows)
    w -= learning_rate * grad_w
    b -= learning_rate * grad_b
    mse = sum((w * row["x"] + b - row["y"]) ** 2 for row in rows) / len(rows)
    emit_progress(step=step, step_target=steps, metrics={"train_mse": mse})

run_dir = Path(os.environ["AR_RUN_DIR"])
run_dir.mkdir(parents=True, exist_ok=True)
(run_dir / "candidate.json").write_text(
    json.dumps({"w": w, "b": b}) + "\n", encoding="utf-8"
)
print(f"train_mse={mse:.12f}")

eval.py is frozen and prints the only objective line:

from __future__ import annotations

import json
import os
from pathlib import Path


def read_jsonl(path: Path) -> list[dict[str, float]]:
    return [json.loads(line) for line in path.read_text(encoding="utf-8").splitlines()]


model = json.loads((Path(os.environ["AR_RUN_DIR"]) / "candidate.json").read_text())
rows = read_jsonl(Path("data/eval.jsonl"))
mse = sum((model["w"] * row["x"] + model["b"] - row["y"]) ** 2 for row in rows) / len(rows)
print(f"eval_score={-mse:.12f}")

Create program.md with the mutation contract:

# Optimisation contract

Minimise validation MSE. Only train.py is mutable. Preserve emit_progress and the
candidate artifact at $AR_RUN_DIR/candidate.json. Never read data/eval.jsonl.

Use absolute paths for workdir and the interpreter. The interpreter must contain autoresearch_rl because train.py imports emit_progress.

name: linear-regression-guide

objective:
  metric: eval_score
  direction: max

target:
  type: command
  workdir: /absolute/path/to/project
  timeout_s: 30
  prepare_cmd: ["/absolute/path/to/project/.venv/bin/python", "prepare.py"]
  train_cmd: ["/absolute/path/to/project/.venv/bin/python", "train.py"]
  eval_cmd: ["/absolute/path/to/project/.venv/bin/python", "eval.py"]

policy:
  type: grid
  params:
    learning_rate: [0.01, 0.05]
    steps: [20, 80]
  mutable_file: /absolute/path/to/project/train.py
  frozen_file: /absolute/path/to/project/prepare.py
  program_file: /absolute/path/to/project/program.md
  required_calls: ["emit_progress"]

controller:
  max_iterations: 4
  max_wall_time_s: 120
  checkpoint_path: /absolute/path/to/project/artifacts/checkpoint.json

comparability:
  expected_budget_s: 120
  strict: false

telemetry:
  ledger_path: /absolute/path/to/project/artifacts/results.tsv
  trace_path: /absolute/path/to/project/traces/events.jsonl
  artifacts_dir: /absolute/path/to/project/artifacts/runs
  versions_dir: /absolute/path/to/project/artifacts/versions

Validate before execution, then read persisted state rather than trusting console memory:

./.venv/bin/autoresearch-rl validate config.yaml
./.venv/bin/autoresearch-rl run config.yaml
./.venv/bin/autoresearch-rl status config.yaml --last 4

validate prints OK, and run prints its result as JSON on stdout:

{
  "iterations": 4,
  "best_value": -0.000182121713,
  "best_score": 0.000182121713
}

best_score is the internal negated form, so it is positive here while best_value keeps the original sign. Every one of the four grid points improved on its predecessor, so the ledger holds four keep rows and the winning point is written to artifacts/versions/v0003/version.json:

{
  "iter": 3,
  "metrics": { "eval_score": -0.000182121713, "train_mse": 2.6457677133367482e-05 },
  "params": { "learning_rate": 0.05, "steps": 80 },
  "status": "ok"
}

Two details of this configuration are load-bearing rather than cosmetic. comparability is pinned because strict defaults to true and the engine compares max_wall_time_s against comparability.expected_budget_s; setting a wall-time budget alone raises ValueError: Non-comparable run blocked: budget_mismatch:120!=300 before the first trial. And the example caps the campaign at four points deliberately: the campaign forecaster begins at five and routinely stops a pure parameter search before a grid is exhausted, as the next section measures.

3. Read the right artifacts

Artifact Read it for
artifacts/results.tsv Per-iteration objective, decision, episode, budget mode, hardware fingerprint, comparability
artifacts/versions/vNNNN/version.json Kept parameters, metrics, run directory, optional model directory, optional full diff
artifacts/checkpoint.json Resume state, best score/value, history, failure window, elapsed time, cost snapshot
artifacts/runs/run-NNNN/ Candidate files, progress JSONL, control file, per-run manifests
traces/events.jsonl Proposals, progress, iteration results, cost events, episode summary
traces/timeline.json Optional Chrome-trace spans for proposer, executor, LLM, and Basilica phases

status reads checkpoint history and includes stderr_tail, so it needs controller.checkpoint_path; without that field it reports iterations_done: 0 and an empty history even after a completed campaign. A clean process exit with no objective metric is changed to internal status failed while the decision stays discard, so the ledger's status column shows discard and its metric column shows 0.000000. Use the stderr_tail and stdout_tail before changing controller logic.

4. Put RL inside the trial deliberately

For GRPO, train.py owns model loading, grouped rollouts, reward shaping, optimiser steps, and candidate saving. eval.py owns held-out pass rate or reward. The outer configuration then searches choices such as learning rate, group size, number of generations, temperature, KL coefficient, and maximum steps.

The supplied source has a basilica-grpo recipe for Qwen2.5-0.5B on GSM8K. Treat it as a starting point, not a current production pin: its dependency and image versions belong to the source snapshot and need a compatibility revalidation. The correct nesting is:

objective:
  metric: eval_score       # frozen held-out score, not mutable training reward
  direction: max
target:
  type: basilica
  prepare_cmd: ["python3", "/app/prepare.py"]
  train_cmd: ["python3", "/app/train.py"]
  eval_cmd: ["python3", "/app/eval.py"]
policy:
  type: hybrid
  params:
    learning_rate: [0.000003, 0.000005, 0.00001]
    max_steps: [30, 50, 80]
    num_generations: [2, 4]
  mutable_file: /absolute/repo/examples/my-grpo/train.py
  frozen_file: /absolute/repo/examples/my-grpo/prepare.py
  program_file: /absolute/repo/examples/my-grpo/program.md

Start with grid or random under the same GPU-hour and iteration budget. Only then measure whether llm or hybrid improves best-of-budget score, failure rate, and proposer cost.

5. Understand the outer learned policy before selecting it

The learned policy is a separate PPO agent over experiment choices. All of the following is in public 0.4.0 and can be read from the installed package:

  • State is an 11-vector: the last eight metric values, a consecutive-keep count, a recent failure count, and history length divided by 100.
  • Action is one element of the full Cartesian parameter space, so the action dimension is the product of the parameter list lengths.
  • Reward is 1.0 for keep, 0.0 for an ordinary discard, -0.05 for cancellation, and -0.1 for failure. That mapping is fixed in the engine and is not configurable.
  • GAE and a clipped PPO loss update after eight transitions by default, over four epochs.
  • Action selection is int(np.argmax(probs)), not sampling. Gradients are central finite differences taken one parameter at a time over every actor and critic weight and bias, so each SGD step costs two full batch evaluations per parameter.
  • PPOAgent seeds its actor and critic with a hard-coded np.random.default_rng(42), so neither policy.seed nor controller.seed changes its initialisation.
  • Learned-policy hyperparameters are not exposed in PolicyConfig; the dispatcher constructs LearnedSearchConfig() defaults.

This is an inspectable research implementation, not a scalable experiment optimiser. Metric features are fed in raw and unnormalised, there is no exploration at action selection, finite-difference cost grows with the product of network size and action count, and the state uses next(iter(metrics.values())), which is the first metric the trial happened to emit and need not be the configured objective. Keep the action space tiny and compare it against seeded random search.

How to develop with it

Author a recipe

  1. Define the candidate artifact and evaluator API first. The evaluator must fail closed on a missing, malformed, NaN, infinite, or incompatible artifact.
  2. Write prepare.py and eval.py. Add corruption, boundary, data-leak, and reference-equivalence tests before writing the mutable trainer.
  3. Write train.py against AR_PARAMS_JSON, AR_RUN_DIR, and optionally AR_MODEL_DIR. Emit progress at a stable cadence.
  4. Write program.md with allowed edits, fixed interfaces, metric direction, resource limits, and forbidden evaluator access.
  5. Add mutable_file, frozen_file, program_file, and meaningful dotted required_calls to the policy config.
  6. Run the command target end to end. Then use one paid run-one smoke before a bounded cloud campaign.

required_calls only protects calls that existed before the patch. If the pre-patch source contains no named call, its absence after the patch is accepted. A call-presence check also proves neither semantics nor provenance: keeping emit_progress(...) does not prove that the reported metric is honest.

For transport-independent RL recipes, keep rollout and environment access behind small interfaces so local tests can bind in-process implementations and a cloud recipe can bind HTTP clients. The private snapshot uses this pattern for disaggregated rollout and tool environments, injecting endpoints as AR_VLLM_ENDPOINT and AR_ENV_ENDPOINT; neither variable exists in public 0.4.0, so on the public package you own that plumbing yourself.

Extend a policy

A parameter policy implements propose(state) -> ParamProposal; a diff policy returns DiffProposal. A batched parameter policy may add propose_batch(state, k). A learnable policy adds record_reward(reward).

Both are typing.Protocols, so structural typing is enough and there is no base class to inherit. What the protocol does not give you is YAML selection: _policy_from_config is a closed if chain over the literal type names, and an unrecognised policy.type silently resolves to StaticPolicy, which proposes an empty parameter dict. On the public package there is no source to patch, so a custom policy has two usable entry points: import run_continuous and pass the object in, or drive run-one from your own loop and own the stopping rule as well.

If you do hold the source, YAML selection additionally requires adding the name to PolicyConfig.type, validating its required fields in config.py, constructing it in controller/continuous.py::_policy_from_config, deciding serial or parallel routing, and adding checkpoint state if the learning must survive a resume. Nothing currently persists policy state, so the last item is new work rather than a hook.

Test malformed provider output, values outside the allowed search space, empty parameter lists, batch-size mismatches, repeated proposals, and completion-order changes under parallel execution.

Extend a target

A target implements run(run_dir, params) and eval(run_dir, params), returning status, metrics, stdout, stderr, elapsed seconds, and run directory. Optional structural methods provide dataset/model injection, setup/teardown, resource cost, and live endpoints.

A YAML-selectable target also needs a TargetConfig.type literal and a constructor branch in target/registry.py. Test these cases:

  • preparation fails before training;
  • training exits nonzero;
  • evaluation exits zero without the objective;
  • timeout and partial logs;
  • NaN and infinity rejection in the evaluator wrapper;
  • concurrent trials receive distinct run, progress, and control paths;
  • teardown runs after exceptions and signals;
  • the returned metric belongs to the candidate just trained, not another worker.

The HTTP target sends separate {"mode":"train"} and {"mode":"eval"} requests and trusts the returned metrics object. Authentication, idempotency, candidate identity, and artifact binding belong in the service contract.

Run the adversarial core probe

The following block runs against the public 0.4.0 install described above, so any reader can reproduce it. It checks GAE against an independent backward recursion, characterises the campaign forecaster on four curves, exercises three metric-parser traps, and probes what the diff screen actually rejects.

from __future__ import annotations

import sys
import tempfile
from pathlib import Path

import numpy as np

from autoresearch_rl.forecasting import forecast_value, should_early_stop
from autoresearch_rl.policy.gae import compute_gae, compute_returns
from autoresearch_rl.sandbox.ast_policy import validate_python_source
from autoresearch_rl.sandbox.validator import validate_diff
from autoresearch_rl.target.command import CommandTarget


def reference_gae(rewards, values, next_value, gamma=0.99, lam=0.95):
    out = [0.0] * len(rewards)
    running = 0.0
    for index in reversed(range(len(rewards))):
        following = next_value if index == len(rewards) - 1 else values[index + 1]
        delta = rewards[index] + gamma * following - values[index]
        running = delta + gamma * lam * running
        out[index] = running
    return out


# 1. GAE, against an independent backward recursion.
rewards = [1.0, 0.0, -1.0]
values = [0.3, 0.2, 0.1]
advantages = compute_gae(rewards, values, next_value=0.0)
assert np.allclose(advantages, reference_gae(rewards, values, 0.0))
assert np.allclose(compute_returns(advantages, values), np.add(advantages, values))
assert compute_gae([], [], next_value=0.0) == []

# 2. The campaign forecaster evaluates its fit at the last point it already
#    measured, so the test is whether the log-linear fit overshoots at the tail,
#    not whether headroom remains. It therefore fires on a healthy decreasing
#    curve and stays quiet on one that has flattened.
IMPROVING = [10.0, 6.0, 3.0, 2.0, 1.6]
FLATTENED = [5.0, 4.0, 3.5, 3.3, 3.29, 3.288]
assert forecast_value(IMPROVING, 5) > min(IMPROVING)
assert should_early_stop(IMPROVING, min(IMPROVING)) is True
assert should_early_stop(IMPROVING[:4], min(IMPROVING[:4])) is False   # needs five points
assert should_early_stop(FLATTENED, min(FLATTENED)) is False           # inverted verdict

# 3. should_early_stop is documented for minimisation. The trial-level guard
#    negates the series for direction: max; the campaign-level call does not.
#    Negating repairs the fit on a negative-valued objective but not the verdict.
MAXIMISING = [-10.661102735427178, -6.758481322317483, -1.4142018507221803,
              -0.07718868008534731, -0.011782072470964826]
assert forecast_value(MAXIMISING, 5) > 26 and MAXIMISING[-1] < 0        # unusable fit
assert should_early_stop(MAXIMISING, max(MAXIMISING)) is True
NEGATED = [-value for value in MAXIMISING]
assert abs(forecast_value(NEGATED, 5) - NEGATED[-1]) < 0.03             # fit repaired
assert should_early_stop(NEGATED, min(NEGATED)) is True                 # verdict unchanged

# 4. Command-target metric parsing. The generic key=value fallback runs only when
#    neither canonical regex matched, and JSON is never parsed.
with tempfile.TemporaryDirectory() as temp_dir:
    def metrics_of(script: str, name: str) -> dict:
        target = CommandTarget(
            train_cmd=[sys.executable, "-c", script], eval_cmd=None,
            workdir=temp_dir, timeout_s=10,
        )
        return target.run(run_dir=str(Path(temp_dir) / name), params={}).metrics

    mixed = metrics_of("print('loss=1.0'); print('eval_score=2.0')", "mixed")
    json_only = metrics_of("print('{\"eval_score\": 2.0}')", "json")
    custom = metrics_of("print('eval_score=2.0')", "custom")
assert mixed == {"loss": 1.0}, mixed          # eval_score silently dropped
assert json_only == {}, json_only
assert custom == {"eval_score": 2.0}, custom

# 5. Diff validation is a static screen, not a sandbox. validate_diff parses only
#    the ADDED lines and denies a short list of imports and dotted calls, so the
#    ordinary ways of reaching the network or the frozen evaluator pass.
def added(*lines: str) -> str:
    body = "".join("+" + line + "\n" for line in lines)
    return "--- a/train.py\n+++ b/train.py\n@@ -1,1 +1,%d @@\n%s" % (len(lines), body)

literal_import = validate_diff(added("import socket")).ok
dynamic_import = validate_diff(added("__import__('socket').socket()")).ok
evaluator_write = validate_diff(added("open('prepare.py','w').write('')")).ok
assert literal_import is False                                         # token match
assert dynamic_import is True
assert evaluator_write is True
assert validate_diff(added("import importlib",
                          "importlib.import_module('subprocess')")).ok is True
assert validate_diff(added("import os", "os.popen('id')")).ok is True

# The stricter whole-source policy exists in the same package and is what the
# added-lines screen approximates, but nothing in the executor feeds it the
# post-patch source, so its extra reach is never exercised on a real diff.
assert validate_python_source("import urllib").ok is False
assert validate_diff(added("import urllib")).ok is False
assert validate_python_source("__import__('socket')").ok is True       # same blind spot

print(
    f"GAE={np.round(advantages, 6).tolist()} | "
    f"forecast stops improving/flattened="
    f"{should_early_stop(IMPROVING, min(IMPROVING))}/"
    f"{should_early_stop(FLATTENED, min(FLATTENED))} | "
    f"maximising fit@5={forecast_value(MAXIMISING, 5):.2f} vs {MAXIMISING[-1]:.6f}, "
    f"negated={forecast_value(NEGATED, 5):.6f} vs {NEGATED[-1]:.6f} | "
    f"metrics mixed={sorted(mixed)} json={sorted(json_only)} custom={sorted(custom)} | "
    f"diff literal/dynamic import accepted={literal_import}/{dynamic_import}, "
    f"evaluator overwrite accepted={evaluator_write}"
)

Executed output:

GAE=[-0.169985, -1.13555, -1.1] | forecast stops improving/flattened=True/False | maximising fit@5=26.43 vs -0.011782, negated=0.034972 vs 0.011782 | metrics mixed=['loss'] json=[] custom=['eval_score'] | diff literal/dynamic import accepted=False/True, evaluator overwrite accepted=True

Three readings follow.

The forecaster's verdicts are inverted against intent. It fits y = a*x**b + c and evaluates the fit at the last step it has already measured, so the question it answers is whether the log-linear fit overshoots at the tail, not whether headroom remains. The offset c is grid-searched over exactly three candidates, 0.5, 0.8, and 0.9 times min(y), so unless a curve's asymptote happens to be one of those three ratios of its own minimum the fit cannot pass through the data. A still-improving curve is stopped; a flattened one is not.

The same forecaster is direction-corrected in one place and not the other. IntraIterationGuard.evaluate negates both the series and the target when objective.direction is max, with a comment stating that the forecaster assumes minimisation. The campaign-level call in the engine passes raw values. On a negative-valued objective that also drives every candidate offset above the series, the pre-log clamp replaces points, and the fit becomes unusable: 26.43 against an observed -0.011782. Negating repairs the fit without changing the stop, because the tail overshoot is the dominant cause.

Diff validation is a static screen, not a sandbox. A literal import socket is caught by a substring token guard. __import__('socket'), importlib.import_module('subprocess'), os.popen, and open('prepare.py','w') all pass. The whole-source AST policy in the same package is stricter, but the executor only ever hands it the added lines of a patch, and it shares the dynamic-import blind spot in any case. Run diff trials in an external sandbox with no secrets, a deny-by-default network, resource limits, and a narrow artifact export.

How to maintain it

  • Pin public deployments to autoresearch-rl==0.4.0. Pin source deployments to a full commit and archive the resolved config plus lockfile. The source snapshot's unchanged package version does not identify its capabilities.
  • Query installed version with importlib.metadata.version("autoresearch-rl") or the CLI. Do not branch on the public release's stale module attribute.
  • Keep output paths unique per campaign. Reusing a checkpoint resumes old history; reusing a ledger mixes episodes; reusing a mutable file carries accepted diffs into the next campaign.
  • Set max_iterations to the intended number of unique grid points. GridPolicy cycles rather than reporting exhaustion.
  • Use an explicit eval_cmd on command targets. Without it, CommandTarget.eval() calls run() again, so one logical iteration trains twice. Basilica caches the training outcome instead and does not deploy twice.
  • Test metric output with the target actually used. The two parsers differ in ways that make the same trial output work locally and return nothing in the cloud. The command target's fallback splits on the first = and tolerates spaces; Basilica scans for (\w+)=([\d.eE+-]+), which requires no space before =, lowercases the key, and returns an empty dict unless one of eval_score, val_bpb, loss, accuracy, f1, training_seconds, improvement, or reward is present. Probed directly, eval_score=2.0 parses on Basilica and eval_score = 2.0 does not. Print the objective without spaces around the sign.
  • Keep generated model artifacts in AR_RUN_DIR or AR_MODEL_DIR, never in one shared filename when parallel mode is enabled.
  • Rotate trace and ledger files deliberately. Rotation is size-based and retains a configured finite number of copies.

For the supplied checkout, use the repository's own ladder before changing core code:

uv sync --extra dev
make validate CONFIG=examples/minimal-trainable-target/config.yaml
make smoke
uv run --no-sync pytest -q
make check

Add a realistic end-to-end test for the changed path. Unit tests did not catch the scaffold and executor gaps described on this page.

The public sdist carries its own suite but not the fixtures it needs. Running it gives 605 passed, 41 failed, 6 skipped; all 41 failures land in test_examples_smoke, test_model_flag, test_scaffold_init, test_cli_init, test_loop_autonomy, and test_scaffold, and every cause is a missing example path reported as FileNotFoundError, Unknown example: ..., or contract_file_missing:prepare.py. Treat the 605 as the meaningful signal and do not read the 41 as library defects.

The declared dependency typer>=0.12 is also too broad to identify a tested CLI combination. In a private-snapshot archive resolved against Typer 0.24.1, a CLI test failed because its fallback imports typer._click, which that release does not export, while the normal CLI worked once Click 8.4.1 was installed. Pin and test Typer together with Click, including the intended no-standalone-Click fallback, before publishing a source build.

Scaffolding caveats

Public 0.4.0 cannot scaffold because no example directory is packaged. The supplied source can copy the bundled minimal example, but at the audited commit it rewrites program_path and policy file paths while leaving target.workdir: examples/minimal-trainable-target. From a scaffold outside the repository, that path is stale. Set target.workdir to the scaffold directory before validation and execution.

The bundled minimal example defaults to llm_diff, so validation also fails until the configured provider key is present or the policy is edited to an offline grid/random baseline. validate does not accept policy overrides; change the copied config first.

How to run it in production

Bounded campaigns

Use this promotion sequence:

  1. Run validate with the exact dataset and environment.
  2. Run one command-target trial and assert the candidate artifact plus frozen evaluator output.
  3. Run run-one on the paid target with an explicit parameter set.
  4. Run a bounded grid or seeded-random baseline.
  5. Run the LLM or diff policy at the same trial and GPU-hour budget.
  6. Replay the selected winner from its version record in a fresh directory and independent evaluator process.
  7. Promote only if the replay passes held-out, regression, safety, latency, and cost gates.

Production configuration needs explicit termination bounds. max_gpu_hours projects from observed Basilica iteration cost and therefore cannot prevent the first trial from overshooting a small budget. It is rejected when controller parallelism is enabled because that cost accounting is not implemented. Pair it with wall time, iteration count, provider-side quotas, and deployment TTL.

Parallel mode applies only to parameter policies. Diff and hybrid campaigns remain serial because concurrent patches would race on one mutable file. The resource pool controls integer capacities but has no fairness layer.

Cooperative cancellation is opt-in and has three preconditions the configuration does not check for you. The guard accumulates progress reports for the exact objective.metric, so merely calling emit_progress is insufficient: the runtime validator checks that a call exists, not that its metric dictionary carries the objective. The guard is armed only once a best value exists, so the first iteration can never be cancelled. And it needs at least five reports and min_steps worth of steps before it may decide. The local guide emits train_mse, not frozen eval_score, so cancellation is intentionally disabled there. Enable it only when an honest progressive objective is available without moving the release gate into mutable code, and only with an explicit eval_cmd: without one the command target executes the trial twice into the same progress.jsonl, so the series the guard forecasts restarts from the beginning halfway through.

When it does fire, the sequence is a contract rather than a kill. The guard writes control.json, the trial's next emit_progress call exits with code 42, and the engine relabels the outcome cancelled, keeping whatever partial metric the last report carried. An executed three-iteration campaign cancelled iterations 1 and 2 with reason forecast_above_best and recorded their mid-flight metric. cancelled is excluded from the failure-rate window but still increments no_improve_streak, so an aggressive cancel policy can end a campaign through no_improve_limit.

Source-only serve-while-learn campaigns

The online surface is not in PyPI 0.4.0. At the supplied source snapshot, it runs a persistent vLLM fleet, records scored experience, trains candidate rounds, evaluates held-out and canary scores, applies forgetting and distribution-shift checks, and may hot-swap a promoted adapter or sparse weight delta.

Adopt it in stages:

  1. Shadow: run the complete collection, training, and gate path while withholding every serving update.
  2. Human-gated: arm campaign.require_approval, set a finite approval timeout and campaign bound, and require an operator decision for each gate-passed candidate.
  3. Automatic promotion: remove the interlock only after shadow and gated evidence cover rollbacks, alarms, stale serving, teardown, and recovery.

The source reference path is:

uv run --no-sync autoresearch-rl validate examples/<recipe>/config.yaml
uv run --no-sync autoresearch-rl campaign run examples/<recipe>/config.yaml --detach
uv run --no-sync autoresearch-rl campaign status <workdir>
uv run --no-sync autoresearch-rl campaign report <workdir>
uv run --no-sync autoresearch-rl slo --workdir <workdir>
uv run --no-sync autoresearch-rl campaign stop <workdir>

Read these production artifacts:

  • result.json for stop reason, round history, promotions, GPU-hours, serving counters, and alarm results;
  • promotions.tsv for gate arithmetic and keep/promote decisions;
  • experience.jsonl for scored served traffic and model version provenance;
  • served_info_boot.json, version snapshots, and served_info_final.json for in-place serving identity;
  • approval_requests.jsonl, approvals.jsonl, and approval_run_id.txt when human gating is armed;
  • deploy_names.json for teardown and straggler verification.

The approval mechanism records accountability, not authentication. Anyone able to write the work directory can write decision records, and the audited source documentation states that the rollout worker's weight-update endpoint has no authentication. Put the controller, artifacts, buckets, and serving control endpoint behind external identity and network policy. Confirm the assembled child command in run.log; config overrides and raw online arguments can change boolean interlocks.

Treat stop_reason values for budget, wall time, or maximum rounds as expected bounded outcomes. Treat forgetting, distribution shift, stale serving, a fatal recorder, or failed teardown as reliability failures. slo exits nonzero on a measured failure; unavailable telemetry reports N-A and must not be presented as a pass.

Failure modes

  • A parameter campaign stops after five results. Campaign forecasting fits a power law and evaluates it at the last already observed point. An executed six-point grid stopped after five with max_iterations set to 6, 7, and 12 alike, on both a direction: max objective and the same sweep restated as direction: min. The lost sixth point scored about 4,300 times better than the best the campaign found. There is no YAML switch for pure parameter policies, and resume does not help because policy state is not checkpointed, so a resumed grid replays the points it already ran. What does work: keep the grid at four points or fewer, drive explicit run-one calls from your own loop, or enable controller.parallel, whose engine never calls the forecaster. Re-running the same six-point grid at max_concurrency: 2 completed all six and returned the best value the serial run had lost. Hybrid and diff routing also disable campaign forecasting. Note that the parallel escape and the GPU budget are mutually exclusive: validate refuses any config that sets max_gpu_hours with parallelism enabled, so on a paid target you trade one guard for the other and must bound spend with wall time, iteration count, and provider quotas instead.
  • The objective disappears when loss is also printed. CommandTarget first parses val_bpb and loss. Its generic key=value fallback runs only when neither was found. Output containing both loss=... and a custom eval_score=... returns only the canonical metric. Print only the objective from eval.py or put it in the final progress report without a canonical stdout metric.
  • JSON metrics produce no result. The command target does not parse {"eval_score": 0.8}. Use one numeric key=value line. Basilica also expects numeric key=value tokens and recognises only a fixed set of completion keys before declaring success.
  • The frozen evaluator is only a label. policy.frozen_file constrains named diff paths when the contract is built. It does not make the file read-only, hide evaluation data, prevent subprocess reads, or validate a score emitted by mutable code.
  • Generated code escapes the intended safety policy. Public 0.4.0 has token checks. The supplied source contains a fuller AST policy, but the active diff executor invokes the token-only form. Run diff trials in an external sandbox with no secrets, a deny-by-default network, resource limits, and a narrow artifact export.
  • emit_progress exists but cancellation never fires. The call reports a proxy metric or wrong key, reports fewer than the configured minimum points, or never observes the control file between long steps.
  • A local trial trains twice. No eval_cmd makes command-target evaluation call the training command again. Define a separate evaluator.
  • A copied source example runs files from the repository. init --target-dir leaves the original relative target.workdir. Replace it with the new absolute directory.
  • A public scaffold has no examples. The 0.4.0 wheel and sdist omit bundled examples even though the command is present.
  • An LLM campaign is secretly random. Parameter LLM policies fall back to seeded random on HTTP failures, invalid JSON, wrong batch count, or a value outside the allowed space. A missing key is caught by validate and blocks the run, but a runtime failure is not: against a closed port, one proposal spent 240.0 s in 10, 20, 40, 80, 90 second backoff and then returned a random draw. Nothing distinguishes it in any artifact. The proposal's rationale field is set to llm-fallback-random but is never persisted; the trace's proposal event carries only params, and the timeline span carries only the iteration and policy class. Capture the controller's stderr, where the fallback is logged as a warning.
  • The learned proposer never explores. Its action method selects argmax; equal initial seeds and state sequences repeat deterministically. The finite-difference optimiser becomes expensive before the action space becomes useful.
  • A paid run exceeds its first-trial budget. GPU-hour projection needs an observed iteration rate. Provider quotas, TTL, and a conservative first run-one remain necessary.
  • Resume mixes an old experiment into a new one, and repeats it. A reused checkpoint restores episode ID, best value, history, failure window, elapsed time, and cost. It does not restore policy state: policy_state is written as an empty dict and never read back, so a resumed grid or seeded-random policy starts its sequence over and re-proposes points it already ran, which then land as discard. score_history is not checkpointed either, so the forecast counter also resets. Use a fresh output tree for a new hypothesis, and do not treat resume as a way to extend a truncated sweep.
  • resume cannot find the run. It expects run-manifest.json in the directory it is given, and that file is written to telemetry.artifacts_dir, not next to checkpoint_path. With the layout above the correct invocation is autoresearch-rl resume artifacts/runs; pointing it at artifacts/ fails with No run-manifest.json in ....

References

  • autoresearch-rl 0.4.0 package, release history, metadata, and downloadable wheel/sdist: https://pypi.org/project/autoresearch-rl/
  • Schulman et al., Proximal Policy Optimization Algorithms: https://arxiv.org/abs/1707.06347
  • Schulman et al., High-Dimensional Continuous Control Using Generalized Advantage Estimation: https://arxiv.org/abs/1506.02438
  • Shao et al., DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models (GRPO): https://arxiv.org/abs/2402.03300
  • Basilica GPU cloud documentation: https://docs.basilica.ai/
  • Hugging Face Hub model upload documentation: https://huggingface.co/docs/hub/models-uploading
  • Perfetto trace viewer, compatible with the optional timeline export: https://ui.perfetto.dev/

Related: Autonomous experimentation loops · Evaluation integrity and anti-gaming · Learning-curve extrapolation and early stopping · Post-training system map · GRPO · PPO · RLVR · Reward design for RL · Asynchronous RL systems · Delta weight synchronization · RL library selection · Experiment tracking and model registry