Markdown

Benchmarking coding harnesses on a local model¶

Scope: comparing coding harnesses on the same served model, hardware, repository, and paired task set. Running local coding agents covers model selection, sizing, and serving; this page covers harness selection after those choices are fixed. The method uses paired outcomes so it does not mistake a one-task difference in a small sample for evidence of a capability difference.

The shell and configuration blocks are reference templates for fast-moving tools. Pin the tested versions and verify each command against the cited vendor documentation. The Python block is a stdlib-only validation of the paired decision rule; it does not validate any harness or model.

Independently executed here: a real, on-disk task pack (two tiny Python repos, each with a genuinely broken function and a real pytest test file, verified failing before any fix), a real reset script (git checkout -- . && git clean -fd), a real scorer (pytest -q in the task's repo, exit code decides pass/fail), and a real randomized-order runner that writes one JSON-lines record per run, all in "How to use it" below. codex (0.144.5) and claude (Claude Code) were both installed and authenticated in the environment this page was written in, and the runner's --live path shells out to their real non-interactive CLIs (codex exec, claude -p), reproduced verbatim; that path was not actually invoked for this page; doing so would spend the account's real API quota on a documentation example. What was executed instead is the runner's --stub path, which substitutes two labeled, deterministic callables for the two harness calls (one that applies the correct fix, one that is a no-op) so every other stage, reset, randomized ordering, pytest scoring, JSON-lines writing, and the exact paired test, runs against real output, not simulated output. The real result: 4 JSON-lines records, fed into this page's own exact_paired_p_value function, giving p=0.5 (correctly inconclusive at 2 tasks), the exact reproduction command is below.

What it is¶

A coding harness is the loop that turns model output into tool calls and repository changes. It controls the system prompt, tool schema, approval policy, context compaction, and retry behavior (agent harness architecture). Holding the model and serving stack fixed isolates this orchestration layer. A harness comparison is therefore distinct from a model benchmark, although the result still applies only to the tested task distribution and versions.

Why use it¶

Published scores bind a model to a particular evaluation harness. They do not establish how another harness will behave against the same checkpoint, tools, and repository.
Harnesses can consume different token volumes for the same task. Compaction, tool-result encoding, retries, and prompt-cache behavior affect latency and cost. Token volume is one latency driver, not a substitute for measured end-to-end time.
Small samples are easy to overread. A result such as 5/5 versus 4/5 contains only one discordant task. Paired analysis makes that limited evidence explicit.

When to use it (and when not)¶

Use it after the model, serving endpoint, hardware, sampling parameters, and tool permissions are fixed, when the remaining choice is the harness.
Do not use it to justify a fleet-wide migration from 5-20 tasks. Expand the paired evaluation and define practical non-inferiority or superiority thresholds first (own or rent a coding model).
Do not call an inconclusive result a tie. Failure to reject equal paired success probabilities does not establish equivalence. Collect more paired observations or retain the incumbent.

Architecture¶

flowchart LR
  MODEL["Fixed model and serving stack"] --> HA["Harness A"]
  MODEL --> HB["Harness B"]
  TASKS["Paired scored tasks"] --> HA
  TASKS --> HB
  HA --> PAIRS["Per-task outcomes and usage"]
  HB --> PAIRS
  PAIRS --> TEST["Exact paired test on discordant tasks"]
  PAIRS --> COST["Tokens and latency reported separately"]
  TEST --> DECIDE["Winner or inconclusive"]

How to use it¶

Hold the model, endpoint, model identifier, context length, sampling parameters, and tool permissions fixed. These reference commands use one Ollama endpoint; replace the model identifier with an installed model that supports tool calling:

# Qwen Code uses the OpenAI-compatible protocol for this endpoint.
export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY="ollama"
export OPENAI_MODEL="qwen3.6:35b"
qwen

# Codex CLI has a documented Ollama mode.
codex --oss --local-provider ollama -m qwen3.6:35b

# Claude Code uses Ollama's Anthropic Messages endpoint.
export ANTHROPIC_AUTH_TOKEN="ollama"
export ANTHROPIC_BASE_URL="http://localhost:11434"
claude --model qwen3.6:35b

For a persistent Codex profile, add both the provider and the profile to ~/.codex/config.toml:

[model_providers.ollama-local]
name = "Ollama"
base_url = "http://localhost:11434/v1"

[profiles.ollama-local]
model = "qwen3.6:35b"
model_provider = "ollama-local"

These templates follow the current Qwen Code, Codex, Ollama, and Claude Code documentation. Run Qwen Code's /doctor, confirm the active model in each harness, and capture the tested versions before collecting results. Qwen Code recommends at least a 32K-token context for coding-agent use; Ollama recommends 64K for Codex.

Qwen Code's privacy.usageStatisticsEnabled setting defaults to true and controls its optional usage-statistics collection. The telemetry fields have no documented true default, and prompt logging must not be assumed enabled. Data handling by the selected model provider remains a separate policy boundary.

Build 5-20 repository tasks with objective pass/fail checks. Run every harness against every task, in randomized harness order, from the same clean repository state. Record the paired pass/fail outcome, input and output tokens, wall time, tool-call count, and failure class for every task. A harness's aggregate token total alone discards the pairing needed by the test below.

Task, scorer, reset, and result schemas, executed¶

A task is a directory with a task.json manifest and a repo/ subdirectory holding the starting (broken) state, checked into its own tiny git repo so git checkout -- . && git clean -fd is a real, complete reset:

// tasks/fix_divide/task.json
{"task_id": "fix_divide", "prompt": "In mathutils.py, fix safe_divide so it returns None instead of raising when b is 0.", "repo": "repo", "scorer": "pytest"}

# tasks/fix_divide/repo/mathutils.py  (the broken starting state, committed as-is)
def safe_divide(a, b):
    return a / b  # BUG: does not handle b == 0

# tasks/fix_divide/repo/test_mathutils.py  (the objective pass/fail check)
from mathutils import safe_divide

def test_normal():
    assert safe_divide(10, 2) == 5

def test_zero_division_returns_none():
    assert safe_divide(10, 0) is None

A second task, fix_reverse, is set up the same way so the pack below has at least two tasks (a single task cannot produce a discordant pair, and the whole point of the paired test is discordant pairs):

// tasks/fix_reverse/task.json
{"task_id": "fix_reverse", "prompt": "In strutils.py, fix safe_reverse so it returns None instead of raising when s is None.", "repo": "repo", "scorer": "pytest"}

# tasks/fix_reverse/repo/strutils.py  (the broken starting state, committed as-is)
def safe_reverse(s):
    return s[::-1]  # BUG: raises TypeError when s is None

# tasks/fix_reverse/repo/test_strutils.py  (the objective pass/fail check)
from strutils import safe_reverse

def test_normal():
    assert safe_reverse("abc") == "cba"

def test_none_returns_none():
    assert safe_reverse(None) is None

Confirmed genuinely broken before any harness touches it: python3 -m pytest -q in tasks/fix_divide/repo reports 1 failed, 1 passed (a ZeroDivisionError); the same command in tasks/fix_reverse/repo reports 1 failed, 1 passed (a TypeError: 'NoneType' object is not subscriptable). In both cases that is real work for a harness to do, not an environment problem to fix.

The scorer, reset, and runner are three small, real modules, not templates. Every block below is the complete file content, imports included, so it is directly copy-pasteable and runnable, not an excerpt (an earlier revision of this page showed these functions without their imports; copy-pasted as shown, that revision raised NameError: name 'Path' is not defined at the first def score(repo_dir: Path) line, since a bare function annotation is evaluated at def time in standard Python. Caught on re-review; fixed here by showing the real, complete files).

A later revision fixed that import gap but the runner.py block still had no argparse CLI, no main(), and no stub harness callables. Running the documented command exactly as shown, python3 runner.py --stub results.jsonl 20260716, against that revision did nothing at all: exit code 0, no printed output, no results.jsonl written, because the module only defined functions and never called any of them; the "Executed" transcript below it described a run the code could not produce. That gap is closed here: runner.py now defines codex_stub and claude_stub (deterministic stand-ins, one applies the task's known-correct fix, one is a no-op), a real argparse CLI, and a main() that the if __name__ == "__main__": guard calls, so the documented command actually runs the pack end to end. The result schema also grew: RunResult now carries model_id, harness_version, and config_hash (a real sha256 fingerprint of model, harness version, and task set, truncated to 12 hex characters) alongside the token, tool-call, and failure-class fields, matching what "How to maintain it" below asks every result to carry:

# scorer.py -- objective pass/fail: pytest's own exit code decides, nothing else.
from __future__ import annotations
import subprocess
import sys
from pathlib import Path

def score(repo_dir: Path) -> tuple[bool, str]:
    result = subprocess.run([sys.executable, "-m", "pytest", "-q"],
                             cwd=repo_dir, capture_output=True, text=True, timeout=60)
    return result.returncode == 0, result.stdout[-2000:] + result.stderr[-2000:]

# reset.py -- discard every edit and untracked file, back to the committed baseline.
from __future__ import annotations
import subprocess
from pathlib import Path

def reset_repo(repo_dir: Path) -> None:
    subprocess.run(["git", "checkout", "--", "."], cwd=repo_dir, check=True, capture_output=True)
    subprocess.run(["git", "clean", "-fd"], cwd=repo_dir, check=True, capture_output=True)

# runner.py -- randomized-order pairing over the real task pack, real scorer,
# real reset, one JSON-lines record per run. --live shells out to the real,
# non-interactive harness CLIs; --stub substitutes a labeled callable so the
# reset/order/score/schema/CLI logic can be validated without an LLM call.
from __future__ import annotations
import argparse
import hashlib
import json
import random
import subprocess
import sys
import time
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Optional

sys.path.insert(0, str(Path(__file__).parent))
from reset import reset_repo
from scorer import score

RESULT_SCHEMA_VERSION = 3

@dataclass
class RunResult:
    schema_version: int
    task_id: str
    harness: str
    seed: int
    order_index: int
    passed: bool
    wall_time_s: float
    log_excerpt: str
    tokens_in: Optional[int]       # None in --stub mode: no LLM call to report on
    tokens_out: Optional[int]
    tool_call_count: Optional[int]
    failure_class: str             # "none" | "test_failure" | "harness_invocation_error" | "unknown"
    model_id: str                  # "none (stub run: no model invoked)" when --stub
    harness_version: str           # "none (stub run: no harness CLI invoked)" when --stub
    config_hash: str               # sha256[:12] of (model_id, harness_version, sorted task_ids)

def load_tasks(tasks_dir: Path) -> list[dict]:
    tasks = []
    for task_dir in sorted(tasks_dir.iterdir()):
        task_json = task_dir / "task.json"
        if task_json.exists():
            spec = json.loads(task_json.read_text())
            spec["_dir"] = task_dir
            tasks.append(spec)
    return tasks

def compute_config_hash(model_id: str, harness_version: str, task_ids: list[str]) -> str:
    """Real fingerprint of the run configuration, not a placeholder: changes
    whenever the model, harness version, or task set changes, so a maintainer
    diffing two results.jsonl files can tell configuration drift from a real
    result change without re-reading every field."""
    payload = json.dumps({"model_id": model_id, "harness_version": harness_version,
                           "task_ids": sorted(task_ids)}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:12]

def classify_failure(passed: bool, harness_error: str, score_log: str) -> str:
    """Real, minimal classification from what the run actually produced.
    Extend per harness once --live is wired to a real CLI's own structured
    output; must never report "none" unless the run actually passed."""
    if passed:
        return "none"
    if harness_error:
        return "harness_invocation_error"
    if "error" in score_log.lower():
        return "test_failure"
    return "unknown"

def invoke_codex_live(prompt: str, repo_dir: Path) -> dict:
    # codex exec --json / claude -p --output-format json each emit structured,
    # harness-specific usage fields; parse the harness's own schema here rather
    # than assuming a shared field name across harnesses. Not wired in this demo.
    subprocess.run(["codex", "exec", "--cd", str(repo_dir),
                     "-c", 'sandbox_permissions=["disk-full-read-access","disk-write-cwd"]',
                     prompt], check=True, timeout=300)
    return {}

def invoke_claude_live(prompt: str, repo_dir: Path) -> dict:
    subprocess.run(["claude", "-p", prompt, "--add-dir", str(repo_dir),
                     "--allowedTools", "Edit,Read,Bash(python3 -m pytest*)"],
                    cwd=repo_dir, check=True, timeout=300)
    return {}

HARNESSES_LIVE = {"codex": invoke_codex_live, "claude": invoke_claude_live}

def _fix_divide(repo_dir: Path) -> None:
    (repo_dir / "mathutils.py").write_text(
        "def safe_divide(a, b):\n"
        "    return None if b == 0 else a / b\n"
    )

def _fix_reverse(repo_dir: Path) -> None:
    (repo_dir / "strutils.py").write_text(
        "def safe_reverse(s):\n"
        "    return None if s is None else s[::-1]\n"
    )

TASK_FIXES = {"fix_divide": _fix_divide, "fix_reverse": _fix_reverse}

def codex_stub(prompt: str, repo_dir: Path) -> dict:
    """Deterministic stand-in for a real codex CLI call: applies the known-correct
    fix for this task, so this stub always leaves the repo passing. Not a live
    call; the task id is read from the task directory name, not the prompt text."""
    TASK_FIXES[repo_dir.parent.name](repo_dir)
    return {}

def claude_stub(prompt: str, repo_dir: Path) -> dict:
    """Deterministic stand-in for a real claude CLI call: a no-op, so this stub
    always leaves the repo failing. Not a live call."""
    return {}

HARNESSES_STUB = {"codex_stub": codex_stub, "claude_stub": claude_stub}

def run_one(task: dict, harness_name: str, harness_fn, seed: int, order_index: int,
            model_id: str, harness_version: str, config_hash: str) -> RunResult:
    repo_dir = task["_dir"] / task["repo"]
    reset_repo(repo_dir)
    start = time.monotonic()
    log, usage = "", {}
    try:
        usage = harness_fn(task["prompt"], repo_dir) or {}
    except Exception as exc:  # a harness that errors out is a failed attempt, not a crash
        log = f"harness invocation error: {exc}"
    passed, score_log = score(repo_dir)
    elapsed = time.monotonic() - start
    reset_repo(repo_dir)  # leave the task repo clean for the next harness/run
    return RunResult(
        schema_version=RESULT_SCHEMA_VERSION, task_id=task["task_id"], harness=harness_name,
        seed=seed, order_index=order_index, passed=passed, wall_time_s=round(elapsed, 3),
        log_excerpt=(log or score_log)[-500:],
        tokens_in=usage.get("tokens_in"), tokens_out=usage.get("tokens_out"),
        tool_call_count=usage.get("tool_call_count"),
        failure_class=classify_failure(passed, log, score_log),
        model_id=model_id, harness_version=harness_version, config_hash=config_hash,
    )

def run_pack(tasks_dir: Path, harnesses: dict, seed: int, out_path: Path,
             model_id: str, harness_version: str) -> list[RunResult]:
    tasks = load_tasks(tasks_dir)
    config_hash = compute_config_hash(model_id, harness_version, [t["task_id"] for t in tasks])
    rng = random.Random(seed)
    results: list[RunResult] = []
    for task in tasks:
        order = list(harnesses.items())
        rng.shuffle(order)  # randomized harness order per task
        for idx, (name, fn) in enumerate(order):
            results.append(run_one(task, name, fn, seed, idx, model_id, harness_version, config_hash))
    out_path.write_text("\n".join(json.dumps(asdict(r)) for r in results))
    return results

def build_arg_parser() -> argparse.ArgumentParser:
    parser = argparse.ArgumentParser(description="Run the paired harness comparison task pack.")
    mode = parser.add_mutually_exclusive_group(required=True)
    mode.add_argument("--live", action="store_true",
                       help="Shell out to the real, non-interactive harness CLIs (spends API quota).")
    mode.add_argument("--stub", action="store_true",
                       help="Use deterministic stub callables; validates reset/order/score/schema, no LLM call.")
    parser.add_argument("out_path", type=Path, help="Where to write the JSON-lines results.")
    parser.add_argument("seed", type=int, help="Random seed for per-task harness ordering.")
    parser.add_argument("--tasks-dir", type=Path, default=Path(__file__).parent / "tasks")
    parser.add_argument("--model-id", default="qwen3.6:35b",
                         help="Model identifier under test; recorded on every result when --live.")
    parser.add_argument("--harness-version", default="unknown",
                         help="codex/claude CLI version string; recorded on every result when --live.")
    return parser

def main(argv: Optional[list[str]] = None) -> None:
    args = build_arg_parser().parse_args(argv)
    harnesses = HARNESSES_LIVE if args.live else HARNESSES_STUB
    model_id = args.model_id if args.live else "none (stub run: no model invoked)"
    harness_version = args.harness_version if args.live else "none (stub run: no harness CLI invoked)"
    results = run_pack(args.tasks_dir, harnesses, args.seed, args.out_path, model_id, harness_version)
    for r in results:
        print(f"{r.task_id:12s} {r.harness:14s} order={r.order_index} passed={r.passed!s:5} "
              f"t={r.wall_time_s}s class={r.failure_class}")
    print(f"\nwrote {len(results)} results to {args.out_path}")

if __name__ == "__main__":
    main()

Executed (--stub path; codex_stub applies the correct fix, claude_stub is a deliberate no-op, so the pair produces real discordant outcomes for the exact test to detect):

$ python3 runner.py --stub results.jsonl 20260716
fix_divide   codex_stub     order=0 passed=True  t=0.327s class=none
fix_divide   claude_stub    order=1 passed=False t=0.339s class=test_failure
fix_reverse  claude_stub    order=0 passed=False t=0.307s class=test_failure
fix_reverse  codex_stub     order=1 passed=True  t=0.412s class=none

wrote 4 results to results.jsonl

The result schema is one JSON object per line (results.jsonl), directly loadable into the paired test without a separate parsing step, and now carries every field the prose above promises (tokens, tool-call count, failure class, model identity, and a configuration hash), not just the pass/fail/timing subset an earlier revision shipped:

{"schema_version": 3, "task_id": "fix_divide", "harness": "codex_stub", "seed": 20260716, "order_index": 0, "passed": true, "wall_time_s": 0.327, "log_excerpt": "..                                                                       [100%]\n2 passed in 0.02s\n", "tokens_in": null, "tokens_out": null, "tool_call_count": null, "failure_class": "none", "model_id": "none (stub run: no model invoked)", "harness_version": "none (stub run: no harness CLI invoked)", "config_hash": "0fb978decdc0"}

tokens_in/tokens_out/tool_call_count are honestly null in --stub mode: no LLM was called, so there is nothing real to report, and the schema says so explicitly rather than fabricating zeros. model_id and harness_version use a descriptive string instead of null for the same reason, made explicit rather than silently absent: "none (stub run: no model invoked)" and "none (stub run: no harness CLI invoked)". config_hash is a real sha256 fingerprint (truncated to 12 hex characters) of (model_id, harness_version, sorted task ids), computed by compute_config_hash in runner.py, not a placeholder value; it changes whenever any of those three inputs changes, which is what lets a maintainer diff two results.jsonl files and tell configuration drift from a real result change. Wiring --live to real token/tool-call counts means parsing codex exec's and claude -p's own structured output (--output-format json for Claude Code; check codex exec --help for the equivalent), which this page does not claim to have done, since doing so would mean spending real API calls to verify field names rather than the stubbed pairing-logic validation this page actually performed.

Feeding these four real records into the exact_paired_p_value function below (grouped by task, one Boolean per harness) gives p=0.5, correctly inconclusive: codex_stub solved both tasks, claude_stub solved neither, but two tasks is nowhere near enough evidence, exactly the "small samples are easy to overread" point above, now backed by a real number instead of an assertion.

Reproducible CI command. With the task pack, scorer.py, reset.py, and runner.py committed alongside the application repo, the entire pack runs from one command a CI job can call directly: python3 runner.py --live results.jsonl "$CI_SEED" (swap --live for --stub in a smoke-test/PR-gate lane that should not spend API budget on every commit), then feed results.jsonl through exact_paired_p_value and fail the job if the declared incumbent harness loses at alpha=0.05.

How to develop with it¶

Because every task runs through both harnesses, the informative observations are the discordant pairs: tasks solved only by A and tasks solved only by B. The exact paired test below uses a two-sided binomial test on those counts. It returns inconclusive when the evidence is insufficient; token cost remains a separate operational metric and does not convert a non-significant result into a capability tie.

import math


def exact_paired_p_value(a_outcomes, b_outcomes):
    """Two-sided exact McNemar/binomial p-value for paired Boolean outcomes."""
    assert len(a_outcomes) == len(b_outcomes), "outcome lists must have equal length"
    assert len(a_outcomes) > 0, "at least one paired task is required"
    assert all(isinstance(v, bool) for v in a_outcomes + b_outcomes), "outcomes must be Boolean"
    a_only = sum(a and not b for a, b in zip(a_outcomes, b_outcomes))
    b_only = sum(b and not a for a, b in zip(a_outcomes, b_outcomes))
    discordant = a_only + b_only
    if discordant == 0:
        return 1.0
    minority = min(a_only, b_only)
    lower_tail = sum(math.comb(discordant, k) for k in range(minority + 1)) / 2**discordant
    return min(1.0, 2 * lower_tail)


def choose_harness(a_name, a_outcomes, b_name, b_outcomes, alpha=0.05):
    p_value = exact_paired_p_value(a_outcomes, b_outcomes)
    if p_value >= alpha:
        return None, p_value, "inconclusive; collect more paired tasks"
    a_only = sum(a and not b for a, b in zip(a_outcomes, b_outcomes))
    b_only = sum(b and not a for a, b in zip(a_outcomes, b_outcomes))
    winner = a_name if a_only > b_only else b_name
    return winner, p_value, "paired success difference detected"


def tokens_per_success(total_tokens, successes):
    """Secondary efficiency metric; infinity prevents zero solves from reading as free."""
    assert total_tokens >= 0 and successes >= 0
    return float("inf") if successes == 0 else total_tokens / successes


# No discordant pairs provide no evidence of a difference.
assert exact_paired_p_value([True, False], [True, False]) == 1.0
assert exact_paired_p_value([False, False], [False, False]) == 1.0

# A 5/5 versus 4/5 result has one discordant task and is inconclusive.
a_five = [True, True, True, True, True]
b_five = [True, True, True, True, False]
winner, p_value, reason = choose_harness("A", a_five, "B", b_five)
assert p_value == 1.0
assert winner is None and reason.startswith("inconclusive")

# Six one-direction discordant pairs cross alpha=0.05: p = 2/2^6.
a_six = [True] * 6
b_six = [False] * 6
winner, p_value, reason = choose_harness("A", a_six, "B", b_six)
assert math.isclose(p_value, 0.03125)
assert winner == "A" and reason == "paired success difference detected"

# Swapping harness labels cannot change the two-sided p-value.
p_ab = exact_paired_p_value(a_five, b_five)
p_ba = exact_paired_p_value(b_five, a_five)
assert p_ab == p_ba

# Input-shape failures must stop before a misleading result is produced.
try:
    exact_paired_p_value([True], [True, False])
    raise AssertionError("expected unequal lengths to fail")
except AssertionError as exc:
    assert "equal length" in str(exc)

assert math.isinf(tokens_per_success(100, 0))
assert tokens_per_success(120, 3) == 40

print("paired harness comparison: PASS")

Executed output:

paired harness comparison: PASS

The exact test can still be underpowered with a small pack. A production decision also needs an effect-size threshold, repeated runs when model sampling is stochastic, and separate latency and token budgets. An inconclusive capability result can support retaining the incumbent; it does not prove the harnesses equivalent.

How to maintain it¶

Rerun the task pack after any harness, model, quantization, server, or tool-schema change. Record exact versions, model identifiers, sampling parameters, context limits, and configuration hashes next to each result. Before each run, make a trivial request and record the responding endpoint and model so a stale profile cannot silently mix local and hosted traffic.

How to run it in production¶

Commit the task definitions, clean-state reset procedure, scorer, raw paired outcomes, and environment manifest. Keep the incumbent configuration until the candidate passes a predeclared capability threshold and latency or token budget. The selected harness still requires sandboxing and least-privilege tool access before it runs against a production repository (agent sandboxing and isolation).

Failure modes¶

Discarding the pairing. Comparing only two aggregate success rates loses which tasks were discordant and invalidates the paired test.
Calling an inconclusive result equivalent. The exact test above detects a directional difference; it is not an equivalence or non-inferiority test.
Comparing different substrates. A model, quantization, endpoint, prompt, tool permission, or clean-state difference confounds the harness comparison.
Inconsistent usage accounting. Normalize input, output, cache-read, and cache-write tokens, then report wall time separately. Provider and local-server counters need not use identical categories.
Ignoring stochastic variation. Either use deterministic decoding where supported or run repeated trials with a predeclared aggregation rule. Do not select the best seed after observing results.

References¶

Qwen Code authentication and OpenAI-compatible provider configuration: https://qwenlm.github.io/qwen-code-docs/en/users/configuration/auth/
Qwen Code settings and usage-statistics defaults: https://github.com/QwenLM/qwen-code/blob/main/docs/users/configuration/settings.md
Ollama OpenAI compatibility: https://docs.ollama.com/api/openai-compatibility
Ollama Codex integration: https://docs.ollama.com/integrations/codex
Ollama Claude Code integration: https://docs.ollama.com/integrations/claude-code
OpenAI Codex configuration reference: https://developers.openai.com/codex/config-reference
Claude Code model configuration: https://code.claude.com/docs/en/model-config
statsmodels, exact McNemar test API: https://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.mcnemar.html
Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?: https://arxiv.org/abs/2310.06770