Markdown

autoresearch-rl¶

Scope: the runner that turns autonomous experimentation loops into a running campaign. It splits every experiment into a frozen prepare.py and a mutable train.py, proposes the next trial with a grid, random, LLM, code-diff, hybrid, or learned policy, executes it locally or on cloud GPU, and keeps or discards on a single scalar. This page covers the file contract, the metric protocol, the environment injected into a trial, and the campaign-level guards, all verified by running the published package. The concepts it implements are on autonomous experimentation loops, evaluation integrity, and early stopping; the outer-controller framing is in the post-training system map.

Everything below was executed against the public PyPI release 0.4.0 (uploaded 2026-05-30) in a clean venv: the CLI, the shipped test suite, and a full six-point grid campaign. The source repository is private, so the only publicly resolvable artifact is the PyPI package, and every claim here is anchored to it. The Python block was executed with numpy and cross-checked against the installed implementation.

What it is¶

Every experiment is two scripts that never import each other. They communicate through the filesystem and through the process environment.

prepare.py is frozen. It runs once, produces the data, and defines what counts as correct: answer extraction and reward computation. No policy may modify it. This is the trust boundary that stops a code-writing optimizer from redefining its own score, the structural rule argued for in evaluation integrity.

train.py is mutable. It runs every iteration, reads the prepared data, trains, and prints metrics. In llm_diff and hybrid mode the LLM rewrites this file. It owns the training algorithm, reward shaping, optimizer, and sampling strategy.

Around that sit three choices. Target is where a trial runs: command (local subprocess), http (remote API), or basilica (GPU cloud). Policy is how the next trial is chosen: grid, random, llm (LLM proposes parameters from history), llm_diff (LLM proposes a unified diff to train.py), hybrid (parameters first, diffs once the improvement streak stalls), or learned (PPO over trajectory feedback). Objective is one metric name and a direction.

The package is small and its dependencies are ordinary: numpy, pydantic, pyyaml, typer, Python 3.10 or newer, MIT licensed. Cloud, HTTP, chart, and object-store integrations are extras.

Why use it¶

The evaluator is frozen by construction, not by convention. Most autonomous-experiment scaffolds put the reward function in the same file the agent is allowed to edit. Splitting it into a separate, unmodifiable script is the whole reason the loop can be trusted to score itself.
One config swaps local for cloud. The same config.yaml runs against a local subprocess or a rented GPU by changing target.type, so a campaign can be debugged on a laptop before it spends GPU-hours.
Failed trials are visible, not silent. A trial that exits cleanly but produces no parseable objective metric is reclassified as failed and lands in the ledger with its stderr, rather than being scored zero.
Campaigns are resumable and budgeted. State is checkpointed after every iteration; stop guards cover wall time, iteration count, no-improvement streak, failure rate, and projected GPU-hours.
Doomed trials can be cancelled cooperatively. A trial that calls emit_progress per step can be told to stop via a control file and exit with code 42, which is recorded as cancelled rather than failed.

When to use it (and when not)¶

Use it when the search space is a handful of hyperparameters plus a training script you are willing to let a model rewrite, and when you can express success as one scalar.
Use grid or random first. They are the baseline that tells you whether an LLM proposer is earning its cost, and skipping that comparison is the standard failure of this whole category.
Use hybrid when parameter search plateaus and the remaining gains are algorithmic.
Do not use it as a distributed trainer. It is an outer controller; the inner trial can launch FSDP, GRPO, or anything else behind a command, but the loop itself knows nothing about ranks or weights.
Do not use it where the objective cannot be reduced to one number that prepare.py can compute without the policy's help.
Do not adopt it from source expecting a public repository. Only the PyPI artifact is public, and it lags the private tree.

Architecture¶

flowchart TB
  subgraph LOOP["Controller"]
    POL["Policy: grid / random / llm / llm_diff / hybrid / learned"]
    EXEC["Executor"]
    EVAL["MetricEvaluator: pick objective.metric, negate when direction is max"]
    DEC{"score < best?"}
    GUARD["Stop guards: wall time, iterations, no-improve, failure rate, GPU-hours, forecast"]
  end
  POL -->|"params or unified diff"| EXEC
  EXEC -->|"AR_PARAM_*, AR_RUN_DIR, AR_PROGRESS_FILE"| TRIAL
  subgraph TRIAL["Trial process"]
    PREP["prepare.py (frozen): data + what is correct"]
    TRAIN["train.py (mutable): how to get there"]
    PREP -->|"data files"| TRAIN
  end
  TRIAL -->|"stdout metrics + progress.jsonl"| EVAL
  EVAL --> DEC
  DEC -->|"keep"| VER["artifacts/versions/vNNNN"]
  DEC -->|"discard"| LEDGER["artifacts/results.tsv"]
  VER --> LEDGER
  LEDGER --> GUARD
  GUARD -->|"continue"| POL
  GUARD -.->|"cancel signal"| TRIAL

How to use it¶

Install the public package and write the two scripts. There is no repository to clone.

python3 -m venv .venv && ./.venv/bin/pip install autoresearch-rl   # 0.4.0, MIT
./.venv/bin/autoresearch-rl --help

The trial runs as a subprocess under whatever interpreter train_cmd names, so that interpreter must be the one with autoresearch_rl installed if the script calls emit_progress. Naming a bare python3 picks up the system interpreter and the trial dies with ModuleNotFoundError.

# config.yaml, executed against 0.4.0. Use an absolute interpreter path.
target:
  type: command
  prepare_cmd: ["/abs/path/.venv/bin/python", "prepare.py"]
  train_cmd:   ["/abs/path/.venv/bin/python", "train.py"]
  workdir: "."
objective:
  metric: eval_score
  direction: max
policy:
  type: grid
  params:
    learning_rate: [0.001, 0.01, 0.05]
    steps: [20, 60]
controller:
  max_iterations: 12
  checkpoint_path: artifacts/checkpoint.json
telemetry:
  ledger_path: artifacts/results.tsv
  trace_path: traces/events.jsonl
  artifacts_dir: artifacts/runs
  versions_dir: artifacts/versions

train.py must print the objective metric in a format the parser recognises. The chain is three deep: a regex for val_bpb and loss, then a bare key = value split on any line, then a backfill from the last emit_progress report. JSON is not parsed. A trial that prints only {"eval_score": -8.48} yields empty metrics and the iteration is recorded as failed.

# train.py: the mutable half. Parameters arrive as AR_PARAM_<UPPERCASE>.
import json, os, pathlib
from autoresearch_rl.target.progress import emit_progress

lr = float(os.environ["AR_PARAM_LEARNING_RATE"])
steps = int(os.environ["AR_PARAM_STEPS"])
rows = [json.loads(l) for l in pathlib.Path("data.jsonl").read_text().splitlines()]

w = b = 0.0
for step in range(steps):
    gw = sum(2 * ((w * r["x"] + b) - r["y"]) * r["x"] for r in rows) / len(rows)
    gb = sum(2 * ((w * r["x"] + b) - r["y"]) for r in rows) / len(rows)
    w -= lr * gw
    b -= lr * gb
    mse = sum(((w * r["x"] + b) - r["y"]) ** 2 for r in rows) / len(rows)
    emit_progress(step=step + 1, step_target=steps, metrics={"eval_score": -mse})

print(f"eval_score = {-mse}")          # `key = value`, not JSON

Running that against a 3x2 grid recovers the target law (w to 2.004, b to 1.0) and produces a monotone ledger, then stops:

iter  learning_rate  steps  eval_score    decision
0     0.001          20     -10.661103    keep
1     0.001          60      -6.758481    keep
2     0.01           20      -1.414202    keep
3     0.01           60      -0.077189    keep
4     0.05           20      -0.011782    keep

The sweep stopped after five of its six points. max_iterations of 6, 7, and 12 all produced exactly five iterations. The sixth combination, {learning_rate: 0.05, steps: 60}, scores -2.73e-06, about 4,300 times better than the best the campaign actually found. The next section explains why.

How to develop with it¶

The stop came from the campaign-level forecast guard, and it is worth understanding before you trust a sweep. The engine fits a power law y = a*x^b + c to the score history and stops when the fitted value at the last already-observed step sits above the running best. It is enabled by default and is switched off only for llm_diff and hybrid mode, on the documented reasoning that a flat trajectory in diff mode is the cue to try new code rather than to quit. Pure parameter search therefore always runs with it on, and there is no YAML field to disable it.

# forecast_guard.py: executed model of the campaign-level early-stop guard.
import numpy as np

C_SCALES = (0.5, 0.8, 0.9)          # offset grid, as a fraction of min(y)
FLOOR = 1e-8                        # clamp applied before the log


def fit_power_law(series):
    """Fit y = a*x^b + c over x = 1..N by log-linear regression on (y - c)."""
    y = np.asarray(series, float)
    x = np.arange(1.0, y.size + 1.0)
    logx = np.log(x)
    best = None
    for c in (y.min() * s for s in C_SCALES):
        z = np.log(np.maximum(FLOOR, y - c))
        den = ((logx - logx.mean()) ** 2).sum()
        if den == 0:
            continue
        b = ((logx - logx.mean()) * (z - z.mean())).sum() / den
        a = np.exp(z.mean() - b * logx.mean())
        resid = ((a * x**b + c - y) ** 2).sum()
        if best is None or resid < best[0]:
            best = (resid, a, b, c)
    if best is None:
        raise ValueError("fit failed")
    return best[1], best[2], best[3]


def forecast(series, step):
    a, b, c = fit_power_law(series)
    return a * float(step) ** b + c


def should_early_stop(series, target, min_points=5):
    """Stop when the curve fitted at the LAST OBSERVED step sits above target."""
    if len(series) < max(3, min_points):
        return False
    try:
        return bool(forecast(series, len(series)) > target)
    except ValueError:
        return False


def clamped_points(series):
    """How many points the FLOOR clamp replaces at the winning offset."""
    y = np.asarray(series, float)
    _, _, c = fit_power_law(series)
    return int((y - c <= FLOOR).sum())


# The score history an executed 6-point grid campaign actually produced.
OBSERVED = [-10.66110273542718, -6.758481322317483, -1.4142018507221794,
            -0.07718868008534736, -0.011782072470964826]

# Equivalence to the shipped implementation, when it is importable.
try:
    from autoresearch_rl import forecasting as ref
    POSITIVE = [10.0, 6.0, 3.0, 2.0, 1.6]
    for s in (POSITIVE, OBSERVED, [5.0, 4.0, 3.5, 3.2, 3.1, 3.05]):
        assert np.allclose(fit_power_law(s), ref.fit_power_law(s), rtol=1e-9)
        assert np.isclose(forecast(s, len(s)), ref.forecast_value(s, len(s)), rtol=1e-9)
        assert should_early_stop(s, max(s)) == ref.should_early_stop(s, max(s))
    REFERENCE_CHECKED = True
except ImportError:
    REFERENCE_CHECKED = False

# The offset grid is a fraction of min(y), so it only sits BELOW the series when
# the values are positive. On a negative-valued metric every candidate offset is
# ABOVE the minimum, y - c goes negative, and the clamp silently replaces points.
assert min(OBSERVED) < 0 and all(min(OBSERVED) * s > min(OBSERVED) for s in C_SCALES)
assert clamped_points(OBSERVED) >= 1
assert clamped_points([10.0, 6.0, 3.0, 2.0, 1.6]) == 0     # positive series is clean

# The consequence: the "forecast" of a step already observed is wildly wrong, and
# wrong in sign, so it cannot be compared against the running best.
predicted = forecast(OBSERVED, len(OBSERVED))
actual = OBSERVED[-1]
assert predicted > 26 and actual < 0
assert abs(predicted - actual) > 26
# Shifting the same curve into positive territory removes the clamping but does
# NOT rescue the forecast: the offset grid is built for a decreasing curve, so a
# saturating increase is still fitted badly. The sign is one problem, not the only one.
shifted = [v + 11.0 for v in OBSERVED]
assert clamped_points(shifted) == 0
assert abs(forecast(shifted, len(shifted)) - shifted[-1]) > 9.0

# So the guard fires on the observed campaign at exactly min_points.
assert should_early_stop(OBSERVED, max(OBSERVED)) is True
assert should_early_stop(OBSERVED[:4], max(OBSERVED[:4])) is False   # 4 points: never
for n in (3, 4):
    assert should_early_stop(OBSERVED[:n], max(OBSERVED[:n])) is False

# It is not only a negative-series problem. A smooth decreasing series also trips
# it, because the fitted curve at the last step sits above the observed minimum.
POSITIVE = [10.0, 6.0, 3.0, 2.0, 1.6]
assert forecast(POSITIVE, 5) > min(POSITIVE)
assert should_early_stop(POSITIVE, min(POSITIVE)) is True

# A series with real headroom left is stopped just the same, which is the point:
# the rule compares a fit against the running best, not against a future step.
STILL_IMPROVING = [100.0, 50.0, 25.0, 12.5, 6.25]
assert should_early_stop(STILL_IMPROVING, min(STILL_IMPROVING)) is True

print(
    f"clamped points on observed series={clamped_points(OBSERVED)}/{len(OBSERVED)} | "
    f"forecast@5={predicted:.2f} vs actual {actual:.6f} | stop at n=4/5: "
    f"{should_early_stop(OBSERVED[:4], max(OBSERVED[:4]))}/"
    f"{should_early_stop(OBSERVED, max(OBSERVED))} | decreasing positive series "
    f"also stops: {should_early_stop(POSITIVE, min(POSITIVE))}"
)

Executed output:

clamped points on observed series=2/5 | forecast@5=26.43 vs actual -0.011782 | stop at n=4/5: False/True | decreasing positive series also stops: True

Three consequences for anyone running a parameter sweep on 0.4.0:

The guard cannot fire before the fifth iteration and very often fires exactly on it. Both the observed maximizing series and a well-behaved decreasing positive series trip it at min_points.
The offset grid assumes a positive, decreasing curve. Candidate offsets are min(y) scaled by 0.5, 0.8, and 0.9, which sit below the series only when the values are positive. On a negative metric such as eval_score = -mse they sit above the minimum, y - c turns negative, and the max(1e-8, ...) clamp replaces those points instead of raising: 2 of 5 points in the observed run. The resulting forecast for a step already observed was 26.43 against an actual of -0.0118.
Shifting the metric positive is not a fix. It removes the clamping but the fit is still off by more than 9 on the same curve, because a saturating increase is not what the offset grid was built for.

Until this changes, keep parameter sweeps short enough to finish inside five iterations, or drive them with run-one from your own loop, or use hybrid, where the guard is disabled. Always compare an LLM policy against grid or random on the same budget.

How to maintain it¶

Pin the version and read it from distribution metadata. In the published 0.4.0 artifact, autoresearch_rl.__version__ is hard-coded "0.2.0". The CLI's --version is correct because it reads the installed distribution, as does importlib.metadata.version("autoresearch-rl"). Branching on the module attribute silently targets the wrong release.
Do not expect init to work from a pip install. Neither the wheel nor the sdist ships the examples/ tree, so autoresearch-rl init <example> exits 1 with an empty "Available examples:" list. It fails cleanly and creates nothing, but the scaffolding path is unavailable to anyone who installed from PyPI. Write the two scripts by hand, as above.
The shipped tests need fixtures that are not shipped. Running the sdist's own suite gives 605 passed, 41 failed, 6 skipped, and every failure traces to the missing examples and recipes rather than to library logic. Treat the 605 as the meaningful signal.
Validate before every campaign. autoresearch-rl validate config.yaml runs eight runtime checks (reserved env prefixes, missing files, API keys, GPU models, unwritable directories, budget alignment, and the presence of emit_progress when intra-iteration cancel is on) and exits 2 before any trial starts.
Use status as the first debugging step. autoresearch-rl status config.yaml --last N returns JSON with per-iteration status, decision, params, metrics, and stderr_tail. That field is what identifies a broken trial immediately.

How to run it in production¶

The interesting production settings are the ones that bound spend and stop waste.

Control	Where	What it does
`controller.max_gpu_hours`	config	pre-trial projection against accumulated GPU-hours; halts before deploying the next trial
`controller.max_wall_time_s`, `max_iterations`	config	hard campaign bounds
`controller.no_improve_limit`, `failure_rate_limit`	config	plateau and broken-environment stops; `cancelled` iterations are not counted as failures
`controller.intra_iteration_cancel`	config	cooperative mid-trial cancel via the forecaster; needs `emit_progress` in the trial
`controller.parallel`	config	K concurrent trials admitted by a resource pool; diff and hybrid stay serial
`telemetry.timeline_path`	config	Chrome-trace JSON openable in `chrome://tracing` or Perfetto

Operationally:

Cooperative cancellation is a contract with the trial, not a kill. The engine writes a control file; the trial's next emit_progress call exits with code 42 and the iteration is recorded as cancelled. A trial that never calls emit_progress can never be cancelled this way, which is why validate checks for it.
emit_progress no-ops when AR_PROGRESS_FILE is unset, so the same train.py still runs standalone outside the framework. The corollary is that a misconfigured interpreter produces zero progress reports and no error.
Parallel mode changes proposal semantics. LLMParamPolicy.propose_batch issues one chat call asking for k diverse proposals rather than k independent calls, and reward feedback to learnable policies is buffered and drained in submission order so the learner sees a stable sequence.
Keep contract_strict on and required_calls populated. The diff validator walks the post-patch AST and rejects any diff that removes a required call, which is what keeps load-bearing instrumentation such as emit_progress alive across LLM-authored edits.
Checkpoint to durable storage. State is written after every iteration and resume continues from the last completed one; the campaign is otherwise restartable only from scratch.
Budget alignment is checked, not enforced by the cloud. max_gpu_hours uses the observed per-iteration rate to project the next trial. A first trial that is far more expensive than steady state can still overshoot.

Failure modes¶

A sweep that stops at five iterations. The forecast guard, described and measured above. In the executed campaign it cost the best grid point by a factor of about 4,300.
JSON metric output. The parser understands val_bpb/loss regexes, bare key = value lines, and emit_progress backfill. A trial printing JSON produces empty metrics; the iteration is then marked failed, which is visible in the ledger but easy to misread as a crash.
A bare python3 in train_cmd. The trial runs under the system interpreter, from autoresearch_rl.target.progress import emit_progress raises ModuleNotFoundError, every iteration fails, and the campaign reports best_value: null after burning its iteration budget. status --last shows the traceback in stderr_tail.
Trusting autoresearch_rl.__version__. Reports 0.2.0 from the 0.4.0 release.
Expecting init or the examples. Not packaged; the scaffold path is repository-only and the repository is private.
Running a sweep without a baseline. The framework makes an LLM policy as easy to select as grid, which is exactly how campaigns end up with no evidence that the proposer beat random search.
Editing the evaluator. Moving reward computation from prepare.py into train.py for convenience dissolves the one property that makes the loop's own scores trustworthy.

References¶

Package on PyPI (the only publicly resolvable artifact; 0.4.0, MIT, uploaded 2026-05-30): https://pypi.org/project/autoresearch-rl/
Release history and file listing: https://pypi.org/project/autoresearch-rl/#history
Basilica GPU cloud, the basilica target's backend: https://basilica.ai/
Chrome trace event format, consumed by telemetry.timeline_path: https://docs.google.com/document/d/1CvAClvFfyA5R-PhYUmn5OOQtYMH4h6I0nSsKchNAySU/preview
Perfetto UI, for opening the exported timeline: https://ui.perfetto.dev/