Markdown

Benchmarking coding harnesses on a local model¶

Scope: comparing coding harnesses on the same served model, hardware, repository, and paired task set. Running local coding agents covers model selection, sizing, and serving; this page covers harness selection after those choices are fixed. The method uses paired outcomes so it does not mistake a one-task difference in a small sample for evidence of a capability difference.

The shell and configuration blocks are reference templates for fast-moving tools. Pin the tested versions and verify each command against the cited vendor documentation. The Python block is a stdlib-only validation of the paired decision rule; it does not validate any harness or model.

What it is¶

A coding harness is the loop that turns model output into tool calls and repository changes. It controls the system prompt, tool schema, approval policy, context compaction, and retry behavior (agent harness architecture). Holding the model and serving stack fixed isolates this orchestration layer. A harness comparison is therefore distinct from a model benchmark, although the result still applies only to the tested task distribution and versions.

Why use it¶

Published scores bind a model to a particular evaluation harness. They do not establish how another harness will behave against the same checkpoint, tools, and repository.
Harnesses can consume different token volumes for the same task. Compaction, tool-result encoding, retries, and prompt-cache behavior affect latency and cost. Token volume is one latency driver, not a substitute for measured end-to-end time.
Small samples are easy to overread. A result such as 5/5 versus 4/5 contains only one discordant task. Paired analysis makes that limited evidence explicit.

When to use it (and when not)¶

Use it after the model, serving endpoint, hardware, sampling parameters, and tool permissions are fixed, when the remaining choice is the harness.
Do not use it to justify a fleet-wide migration from 5-20 tasks. Expand the paired evaluation and define practical non-inferiority or superiority thresholds first (own or rent a coding model).
Do not call an inconclusive result a tie. Failure to reject equal paired success probabilities does not establish equivalence. Collect more paired observations or retain the incumbent.

Architecture¶

flowchart LR
  MODEL["Fixed model and serving stack"] --> HA["Harness A"]
  MODEL --> HB["Harness B"]
  TASKS["Paired scored tasks"] --> HA
  TASKS --> HB
  HA --> PAIRS["Per-task outcomes and usage"]
  HB --> PAIRS
  PAIRS --> TEST["Exact paired test on discordant tasks"]
  PAIRS --> COST["Tokens and latency reported separately"]
  TEST --> DECIDE["Winner or inconclusive"]

How to use it¶

Hold the model, endpoint, model identifier, context length, sampling parameters, and tool permissions fixed. These reference commands use one Ollama endpoint; replace the model identifier with an installed model that supports tool calling:

# Qwen Code uses the OpenAI-compatible protocol for this endpoint.
export OPENAI_BASE_URL="http://localhost:11434/v1"
export OPENAI_API_KEY="ollama"
export OPENAI_MODEL="qwen3.6:35b"
qwen

# Codex CLI has a documented Ollama mode.
codex --oss --local-provider ollama -m qwen3.6:35b

# Claude Code uses Ollama's Anthropic Messages endpoint.
export ANTHROPIC_AUTH_TOKEN="ollama"
export ANTHROPIC_BASE_URL="http://localhost:11434"
claude --model qwen3.6:35b

For a persistent Codex profile, add both the provider and the profile to ~/.codex/config.toml:

[model_providers.ollama-local]
name = "Ollama"
base_url = "http://localhost:11434/v1"

[profiles.ollama-local]
model = "qwen3.6:35b"
model_provider = "ollama-local"

These templates follow the current Qwen Code, Codex, Ollama, and Claude Code documentation. Run Qwen Code's /doctor, confirm the active model in each harness, and capture the tested versions before collecting results. Qwen Code recommends at least a 32K-token context for coding-agent use; Ollama recommends 64K for Codex.

Qwen Code's privacy.usageStatisticsEnabled setting defaults to true and controls its optional usage-statistics collection. The telemetry fields have no documented true default, and prompt logging must not be assumed enabled. Data handling by the selected model provider remains a separate policy boundary.

Build 5-20 repository tasks with objective pass/fail checks. Run every harness against every task, in randomized harness order, from the same clean repository state. Record the paired pass/fail outcome, input and output tokens, wall time, tool-call count, and failure class for every task. A harness's aggregate token total alone discards the pairing needed by the test below.

How to develop with it¶

Because every task runs through both harnesses, the informative observations are the discordant pairs: tasks solved only by A and tasks solved only by B. The exact paired test below uses a two-sided binomial test on those counts. It returns inconclusive when the evidence is insufficient; token cost remains a separate operational metric and does not convert a non-significant result into a capability tie.

import math


def exact_paired_p_value(a_outcomes, b_outcomes):
    """Two-sided exact McNemar/binomial p-value for paired Boolean outcomes."""
    assert len(a_outcomes) == len(b_outcomes), "outcome lists must have equal length"
    assert len(a_outcomes) > 0, "at least one paired task is required"
    assert all(isinstance(v, bool) for v in a_outcomes + b_outcomes), "outcomes must be Boolean"
    a_only = sum(a and not b for a, b in zip(a_outcomes, b_outcomes))
    b_only = sum(b and not a for a, b in zip(a_outcomes, b_outcomes))
    discordant = a_only + b_only
    if discordant == 0:
        return 1.0
    minority = min(a_only, b_only)
    lower_tail = sum(math.comb(discordant, k) for k in range(minority + 1)) / 2**discordant
    return min(1.0, 2 * lower_tail)


def choose_harness(a_name, a_outcomes, b_name, b_outcomes, alpha=0.05):
    p_value = exact_paired_p_value(a_outcomes, b_outcomes)
    if p_value >= alpha:
        return None, p_value, "inconclusive; collect more paired tasks"
    a_only = sum(a and not b for a, b in zip(a_outcomes, b_outcomes))
    b_only = sum(b and not a for a, b in zip(a_outcomes, b_outcomes))
    winner = a_name if a_only > b_only else b_name
    return winner, p_value, "paired success difference detected"


def tokens_per_success(total_tokens, successes):
    """Secondary efficiency metric; infinity prevents zero solves from reading as free."""
    assert total_tokens >= 0 and successes >= 0
    return float("inf") if successes == 0 else total_tokens / successes


# No discordant pairs provide no evidence of a difference.
assert exact_paired_p_value([True, False], [True, False]) == 1.0
assert exact_paired_p_value([False, False], [False, False]) == 1.0

# A 5/5 versus 4/5 result has one discordant task and is inconclusive.
a_five = [True, True, True, True, True]
b_five = [True, True, True, True, False]
winner, p_value, reason = choose_harness("A", a_five, "B", b_five)
assert p_value == 1.0
assert winner is None and reason.startswith("inconclusive")

# Six one-direction discordant pairs cross alpha=0.05: p = 2/2^6.
a_six = [True] * 6
b_six = [False] * 6
winner, p_value, reason = choose_harness("A", a_six, "B", b_six)
assert math.isclose(p_value, 0.03125)
assert winner == "A" and reason == "paired success difference detected"

# Swapping harness labels cannot change the two-sided p-value.
p_ab = exact_paired_p_value(a_five, b_five)
p_ba = exact_paired_p_value(b_five, a_five)
assert p_ab == p_ba

# Input-shape failures must stop before a misleading result is produced.
try:
    exact_paired_p_value([True], [True, False])
    raise AssertionError("expected unequal lengths to fail")
except AssertionError as exc:
    assert "equal length" in str(exc)

assert math.isinf(tokens_per_success(100, 0))
assert tokens_per_success(120, 3) == 40

print("paired harness comparison: PASS")

Executed output:

paired harness comparison: PASS

The exact test can still be underpowered with a small pack. A production decision also needs an effect-size threshold, repeated runs when model sampling is stochastic, and separate latency and token budgets. An inconclusive capability result can support retaining the incumbent; it does not prove the harnesses equivalent.

How to maintain it¶

Rerun the task pack after any harness, model, quantization, server, or tool-schema change. Record exact versions, model identifiers, sampling parameters, context limits, and configuration hashes next to each result. Before each run, make a trivial request and record the responding endpoint and model so a stale profile cannot silently mix local and hosted traffic.

How to run it in production¶

Commit the task definitions, clean-state reset procedure, scorer, raw paired outcomes, and environment manifest. Keep the incumbent configuration until the candidate passes a predeclared capability threshold and latency or token budget. The selected harness still requires sandboxing and least-privilege tool access before it runs against a production repository (agent sandboxing and isolation).

Failure modes¶

Discarding the pairing. Comparing only two aggregate success rates loses which tasks were discordant and invalidates the paired test.
Calling an inconclusive result equivalent. The exact test above detects a directional difference; it is not an equivalence or non-inferiority test.
Comparing different substrates. A model, quantization, endpoint, prompt, tool permission, or clean-state difference confounds the harness comparison.
Inconsistent usage accounting. Normalize input, output, cache-read, and cache-write tokens, then report wall time separately. Provider and local-server counters need not use identical categories.
Ignoring stochastic variation. Either use deterministic decoding where supported or run repeated trials with a predeclared aggregation rule. Do not select the best seed after observing results.

References¶

Qwen Code authentication and OpenAI-compatible provider configuration: https://qwenlm.github.io/qwen-code-docs/en/users/configuration/auth/
Qwen Code settings and usage-statistics defaults: https://github.com/QwenLM/qwen-code/blob/main/docs/users/configuration/settings.md
Ollama OpenAI compatibility: https://docs.ollama.com/api/openai-compatibility
Ollama Codex integration: https://docs.ollama.com/integrations/codex
Ollama Claude Code integration: https://docs.ollama.com/integrations/claude-code
OpenAI Codex configuration reference: https://developers.openai.com/codex/config-reference
Claude Code model configuration: https://code.claude.com/docs/en/model-config
statsmodels, exact McNemar test API: https://www.statsmodels.org/stable/generated/statsmodels.stats.contingency_tables.mcnemar.html
Jimenez et al., SWE-bench: Can Language Models Resolve Real-World GitHub Issues?: https://arxiv.org/abs/2310.06770