Markdown

The Harness Effect: orchestration-layer token economics¶

Scope: what a controlled swap of the orchestration layer, holding models and tasks fixed, does to the cost, latency, and token intensity of enterprise agentic tasks. This page covers "The Harness Effect" (Sayed Ali et al., Writer Inc., arXiv:2607.06906, July 2026): its token-economics formalization, the six mechanism families that implement the savings, and the measured results across six foundation models. It is the empirical companion to agentic loop economics (why prefix caching dominates) and loop engineering (the harness moves themselves); this page supplies a controlled measurement of their combined effect.

This is a single industry paper from one vendor (Writer, Inc.) evaluating its own commercial harness against its own frozen baseline; the last author is the company's co-founder and CTO. The paper reports one frozen baseline run and does not publish confidence intervals, p-values, or run-to-run variance for the efficiency deltas. Treat the mechanism inventory as an engineering design to test and the percentages as this study's reported results, not industry constants.

flowchart LR
  TASK["22 locked tasks x 6 models"] --> BASE["Baseline: frozen conventional loop<br/>(49KB monolithic prompt, regex tool parsing,<br/>destructive truncation, polling waits)"]
  TASK --> HARN["Writer Agent Harness<br/>(same models, orchestration swapped)"]
  BASE --> METER["Trace shim: prompt/completion tokens,<br/>tool events, wall-clock, per turn"]
  HARN --> METER
  METER --> PRICE["Pinned price table applied at report build"]
  PRICE --> RESULT["Cost -41%, latency -44%, tokens -38%,<br/>quality 0.78 -> 0.81 (parity)"]

What it is¶

"The Harness Effect" is a harness-swap experiment, not a new inference algorithm. The same 22 enterprise tasks run on the same six foundation models under two orchestration layers. The baseline is a production agent loop frozen on 2026-06-07; the treatment is the Writer Agent Harness at its general-availability configuration. Holding tasks, model routes, judges, and the price table fixed is intended to isolate the orchestration change within this experiment. It does not remove run-to-run variance, judge error, or vendor self-evaluation as threats to validity.

The paper formalizes the bill for a k-turn agent loop as C = sum_i(p_in * T_in_i + p_out * T_out_i) and decomposes input tokens into system prompt, replayed history, tool schemas, retrieval payload, and the user turn. Replaying a growing transcript produces quadratic cumulative input growth under the paper's simplified model; bounded live history produces linear growth. Prompt caching changes price, not token count. The paper models effective input price as p_eff = p_in * (1 - h*(1-kappa)), where h is the cache-read fraction and the study sets the cache-read price multiplier kappa to 0.1. Prefix stability affects h, while actual cache eligibility, retention, and pricing remain provider-specific.

Why use it¶

The orchestration layer, not the model menu, moved the bill most. Under the frozen baseline, switching from the most expensive model (Palmyra X6, $0.25/task) to the cheapest (Qwen 3.6, $0.16/task) saved 36%. Keeping any single model and adopting the harness saved 33% to 61%. On this workload the harness was a bigger cost lever than the entire model menu.
Efficiency moved in the same direction across the tested panel. All six tested models became cheaper in the reported results: -33% to -61% in cost and -33% to -55% in median wall-clock. Six models do not establish model invariance.
Quality did not trade against efficiency. Headline task-completion held at parity (0.78 to 0.81, reported as a wash given n=22, not a real gain). Quality per dollar rose 82% (3.71 to 6.75) and task-completions per million tokens rose 68% (54.9 to 92.0), because the denominator (tokens, dollars) shrank while the numerator (quality) did not.
The reported saving is material at fleet scale. At one million tasks per month, applying the study's rounded blended rates gives $210k/month for the baseline and $120k/month for the harness. That extrapolation assumes the task mix, prices, cache behavior, and completion rates remain representative.

When to use it (and when not)¶

Read this as an argument for auditing your own harness's mechanism coverage, not as a benchmark you can reproduce by installing a specific product. The 41%/44%/38% figures are Writer's measured deltas between their own two orchestration layers; your baseline's inefficiency (or lack of it) sets your own achievable delta.
Orchestration features are not free to expose to a weak model. The one net-new capability in the treatment arm, sub-agent delegation, only cleared a usable reliability threshold (task-completion above roughly 0.85) on the two strongest models in the panel (Palmyra X6, Sonnet 4.6); it actively degraded on Gemini 3.1 and GLM 5.1, and was unreliable on the fast tier. Gate advanced orchestration features by model capability rather than exposing one interface to every model.
All seven quality regressions in the study landed on the three smaller models, concentrated in orchestration-heavy capabilities (tool use via MCP, multi-step Playbooks). If your fleet runs weaker or smaller open-weight models, validate that a richer harness helps them before assuming the frontier-model result transfers.
Do not infer statistical significance from a uniform sign. The paper reports the cost, token, and latency deltas as uniform across its panel, but it does not publish the uncertainty needed to reproduce a significance claim. Quality deltas are explicitly directional at n=22.
This is one vendor's harness compared with one vendor's frozen baseline. Table 1 (below) compares six other widely used agent systems (Claude Code, Claude Cowork, LangGraph, CrewAI, AutoGen/AG2, Hermes Agent) on the same mechanism axes from public documentation, not from head-to-head measurement; do not read it as a benchmark ranking.

Architecture¶

The six mechanism families map onto specific terms of the cost equation: caching and compaction attack the input-token decomposition directly, context offload keeps information available without keeping it in the context window, and zero-token waiting plus failure-spend governance bound the retry and dead-end multiplier that no per-token discount can fix.

flowchart TB
  subgraph PROMPT["Two-zone prompt (Mechanism 1: cache-shape discipline)"]
    STABLE["Byte-stable prefix: tool-schema catalog,<br/>stable system prompt, append-only transcript<br/>(up to 4 cache breakpoints, 1hr TTL)"]
    VOLATILE["Volatile tail: clock, file listings, plan state,<br/>one-shot reminders (rebuilt every turn, never cached)"]
  end
  STABLE -->|"measured 99.9% cache-read rate"| CACHE["Effective input price ~0.1x list (Eq. 4)"]
  PROMPT --> COMPACT["Mechanism 2: structured compaction<br/>at 80% budget -> typed checkpoint<br/>(durable memory, exec summary, requirements, skills)<br/>+ live tail of 4-12 messages"]
  COMPACT --> OFFLOAD["Mechanism 3: context offload<br/>sub-agent firewalls (8KB capped summary),<br/>skill progressive disclosure, tool-output spill to files"]
  OFFLOAD --> WAIT["Mechanism 4: zero-token waiting<br/>durable suspend/resume, WAL journaling,<br/>generation fencing on crash recovery"]
  WAIT --> GOVERN["Mechanism 5: failure-spend governance<br/>typed failure classes, discarded-attempt rollback,<br/>circuit breaker, iteration/parallelism caps"]
  GOVERN --> FLOOR["Mechanism 6: model-agnostic floor<br/>typed route plan, normalized chunk contract,<br/>schema hygiene for weaker models"]
  FLOOR --> RESULT2["-41% cost, -44% latency, -38% tokens,<br/>quality at parity"]

The six mechanism families¶

Cache-shape discipline (the two-zone prompt). Every prompt has an enforced physical shape: a byte-stable prefix (full tool-schema catalog, stable system prompt, append-only transcript) followed by a volatile tail rebuilt every turn. The split is a correctness rule, not an optimization: anything that changes per turn is structurally banned from the prefix, and the cache-marker logic refuses to place a breakpoint at or after the first volatile message. Up to four provider cache breakpoints are pinned, with one-hour retention latched per session. Measured on an identical-prefix call in the harness repository, 7,876 of 7,886 prompt tokens (99.9%) were served as cache reads.
Structured, incremental, cache-aware compaction. At 80% of the model's input budget, older history folds into a typed checkpoint carrying four artifacts: durable memory (decisions, constraints, rejected approaches), an eight-section resumability summary, preserved verbatim user requirements, and skill references. A live tail of the 4 to 12 most recent messages always survives verbatim. Each checkpoint folds the previous one forward so compaction cost stays incremental, summarization runs on a cheaper helper model off the paying loop, and a degraded summary aborts the compaction rather than persisting it.
Context offload. Sub-agents act as context firewalls: a child agent reads or searches in its own context and returns a summary capped at 8KB, with citations on a metadata sidecar the parent never reads. Skills use progressive disclosure (only a name-and-description table sits in the prompt; the full document loads only when invoked). Bulky tool outputs spill to files past 20K characters (head-and-tail preview, full output on disk). Plan and canvas state are event-sourced and re-rendered once per turn rather than duplicated across the transcript.
Zero-token waiting; durability as economics. Waiting on a human answer, an approval, or a background job suspends the run durably at zero token cost and resumes on an event, with no polling turns. Every event is journaled to a write-ahead log before being streamed; crashed or preempted runs resume under generation fencing at the next sequence number, so a crash that would otherwise mean re-buying 40 turns of tokens instead resumes from durable state.
Failure-spend governance. Every failure is classified into a typed class (rate limit, stall, timeout, malformed stream, provider outage, permanent) before any retry decision, and only whitelisted classes fall through to the next provider. Mid-stream failures become discarded attempts: partial output is cleared and no side effects can originate from a discarded attempt. A circuit breaker halts a model that re-issues a byte-identical failing tool call three times; the loop is capped at 50 iterations and tool parallelism at four.
A model-agnostic control path. The route plan is data, and every provider stream normalizes into one chunk contract before the loop consumes it. Native tool calling and schema normalization reduce provider-specific branches. This design explains why the same mechanisms can operate across models; it does not guarantee equal savings or quality effects outside the tested panel.

Core math (runnable): reproducing the paper's own equations and numbers¶

This numpy-only block reproduces the derived Table 3 ratios, tests Eq. (4) at both cache boundaries, and validates the quadratic-versus-linear token-growth example. The paper reports a -41% cost delta, while its rounded $0.21 and $0.12 display values reproduce -42.9%. The paper does not publish enough precision to reconcile those figures, so the code reports both without assigning a cause. Run: python3 harness_economics.py.

# harness_economics.py: runnable checks for the paper's derived metrics,
# effective-input-price model, and simplified transcript-growth model.
# This does not reproduce the underlying benchmark runs or unpublished
# full-precision cost data.
import numpy as np

def quality_per_dollar(Q, C):
    return Q / C

def completions_per_million_tokens(Q, tau):
    return Q * 1e6 / tau

def effective_input_price(p_in, h, kappa=0.1):
    # Eq. (4): p_eff = p_in * (1 - h*(1-kappa)); h = fraction of input tokens
    # served as cache reads, kappa = cache-read price multiplier (~0.1x list).
    return p_in * (1 - h * (1 - kappa))

def naive_replay_tokens(k, S, m_bar, G_bar, R_bar):
    # Eq. (3): full-history replay grows O(k^2) in turn count k.
    return sum(S + (i - 1) * m_bar + G_bar + R_bar for i in range(1, k + 1))

def harness_managed_tokens(k, S, G_bar, delta_bar, R_bar):
    # Caching changes the price of S and G_bar, not their token count.
    return k * (S + G_bar + delta_bar + R_bar)

# --- 1) Reproduce Table 3's blended headline numbers exactly ---------------
Q_base, C_base, tau_base = 0.78, 0.21, 14_200
Q_harn, C_harn, tau_harn = 0.81, 0.12, 8_800

eta_base = quality_per_dollar(Q_base, C_base)
eta_harn = quality_per_dollar(Q_harn, C_harn)
cpm_base = completions_per_million_tokens(Q_base, tau_base)
cpm_harn = completions_per_million_tokens(Q_harn, tau_harn)

assert abs(eta_base - 3.7142857142857144) < 1e-9, eta_base
assert abs(eta_harn - 6.75) < 1e-9, eta_harn
eta_gain = eta_harn / eta_base - 1
assert abs(eta_gain - 0.8175) < 0.001, eta_gain          # paper: "+82%"

assert abs(cpm_base - 54.929577464788736) < 1e-6, cpm_base
assert abs(cpm_harn - 92.04545454545455) < 1e-6, cpm_harn
cpm_gain = cpm_harn / cpm_base - 1
assert abs(cpm_gain - 0.6757) < 0.001, cpm_gain          # paper: "+68%"

cost_delta = C_harn / C_base - 1
tau_delta = tau_harn / tau_base - 1
# The rounded display values reproduce -42.9%, not the paper's -41% headline.
# The source does not expose enough precision to reconcile the difference.
assert abs(cost_delta - (-0.42857)) < 1e-4, cost_delta
assert abs(tau_delta - (-0.38028)) < 1e-4, tau_delta     # paper: "-38%" (this one matches)

# --- 2) Effective input price, Eq. (4): reproduce the measured 99.9% hit rate ---
p_in = 1.0
h_measured = 7876 / 7886                                  # measured cache-read fraction
p_eff_measured = effective_input_price(p_in, h_measured, kappa=0.1)
assert 0.10 < p_eff_measured < 0.102, p_eff_measured       # "roughly a tenth of list"

# Boundary: perfect prefix stability (h=1) -> pays exactly kappa * list price.
p_eff_perfect = effective_input_price(p_in, h=1.0, kappa=0.1)
assert np.isclose(p_eff_perfect, 0.1)

# Adversarial: cache-shape discipline dropped entirely (h=0) -> full list price,
# i.e. the discount collapses to nothing regardless of how good the model is.
p_eff_broken = effective_input_price(p_in, h=0.0, kappa=0.1)
assert np.isclose(p_eff_broken, 1.0)
assert p_eff_broken > p_eff_measured, "dropping cache discipline must raise the effective price"

# --- 3) Eq. (3) vs harness-managed growth: quadratic beats a constant-delta loop ---
k = 12
S, m_bar, G_bar, R_bar = 2000, 1500, 3000, 500             # illustrative token sizes
naive_k = naive_replay_tokens(k, S, m_bar, G_bar, R_bar)
naive_k_half = naive_replay_tokens(k // 2, S, m_bar, G_bar, R_bar)
# Quadratic growth: doubling k more than doubles cumulative naive tokens.
assert naive_k > 2 * naive_k_half, (naive_k, naive_k_half)

delta_bar = 400                                            # bounded live-history tokens per turn
harness_k = harness_managed_tokens(k, S, G_bar, delta_bar, R_bar)
harness_k_half = harness_managed_tokens(k // 2, S, G_bar, delta_bar, R_bar)
# Linear growth: doubling k doubles cumulative harness-managed input tokens.
assert np.isclose(harness_k / harness_k_half, 2.0, atol=1e-9)
assert naive_k > harness_k, "at k=12 naive replay must already exceed the harness-managed loop"

# --- 4) Boundary: k=1 (single turn) has no history to replay; both models agree ---
assert np.isclose(naive_replay_tokens(1, S, m_bar, G_bar, R_bar),
                   harness_managed_tokens(1, S, G_bar, 0, R_bar))

print("harness economics: PASS")
print(f"  eta$ {eta_base:.3f} -> {eta_harn:.3f} ({eta_gain:+.1%}, paper: +82%)")
print(f"  CPM  {cpm_base:.1f} -> {cpm_harn:.1f} ({cpm_gain:+.1%}, paper: +68%)")
print(f"  cost {cost_delta:+.1%} from rounded $0.21->$0.12 (paper headline: -41%; precision unavailable)")
print(f"  tokens {tau_delta:+.1%} (paper: -38%, matches)")
print(f"  p_eff at measured hit rate h={h_measured:.4f}: {p_eff_measured:.4f} x list (paper: ~0.1x)")
print(f"  p_eff boundaries: h=1 -> {p_eff_perfect:.3f}x, h=0 -> {p_eff_broken:.3f}x")
print(f"  naive replay @k={k}: {naive_k:.0f} tok  vs  harness-managed: {harness_k:.0f} tok")

Executed output:

harness economics: PASS
  eta$ 3.714 -> 6.750 (+81.7%, paper: +82%)
  CPM  54.9 -> 92.0 (+67.6%, paper: +68%)
  cost -42.9% from rounded $0.21->$0.12 (paper headline: -41%; precision unavailable)
  tokens -38.0% (paper: -38%, matches)
  p_eff at measured hit rate h=0.9987: 0.1011 x list (paper: ~0.1x)
  p_eff boundaries: h=1 -> 0.100x, h=0 -> 1.000x
  naive replay @k=12: 165000 tok  vs  harness-managed: 70800 tok

How it works¶

The experiment is a paired swap, not a new training or inference technique: both arms execute the same 22 tasks (spanning nine capability areas: model/system awareness, grounding and retrieval, content generation, playbooks, MCP tool use, presentations, voice, image analysis, and sub-agents) on the same six models with the same LLM-judge panel and the same pinned price table. Only the orchestration code differs between arms. Cost is computed at report build by multiplying recorded prompt and completion tokens by a pinned price table, identically for both arms, so the token and latency results are price-independent even though the dollar figures depend on one price snapshot.

How to use it¶

Run a paired harness comparison before adopting any mechanism. Freeze the model revision, serving configuration, task inputs, judge, and price table; execute both harnesses on every task; and retain per-task traces. Compare paired task outcomes first. Compare tokens, latency, and cost only after establishing that the quality result meets the workload's acceptance rule.

How to develop with it¶

Apply the following controls to the harness under test:

Audit the prompt's stable and volatile regions. Check whether the system prompt, tool-schema catalog, and eligible transcript prefix are byte-identical across turns. Timestamps, reordered schemas, and regenerated boilerplate can invalidate prefix-cache reuse. Measure the effect against the provider's actual cache counters and prices.
Replace destructive truncation with structured, incremental compaction before the context overflows, not after: a typed checkpoint (durable memory, resumability summary, verbatim requirements, skill references) plus a small live tail, folding forward rather than re-summarizing from scratch each time.
Firewall delegated work. If you delegate to sub-agents, cap what they return (the paper uses an 8KB capped summary) rather than letting the parent context absorb everything a child agent read.
Classify failures before retrying. Retry only typed transient failures. Bound attempts, discard partial failed generations, and prevent side effects from an attempt that the controller will retry.
Gate advanced features by model tier rather than exposing one interface to all models, following the capability-floor finding (Section 6.5 of the paper): scope down tool catalogs and disable delegation below whatever reliability threshold your own evaluation finds for each model, mirroring the practice in evaluating agents.

How to instrument and measure it in production¶

Instrument before optimizing. Record prompt, completion, cache-read, and cache-write tokens; tool events; retries; discarded attempts; and wall time for every turn. Without per-turn traces, higher token intensity can be misattributed to the model or task.
Track completions per million tokens and quality per dollar next to raw quality, not instead of it. Also retain direct service metrics such as task success, p95 latency, and total cost. See agentic loop economics and agent observability.
Measure cache-hit rate on real sessions, not just aggregate token counts; the paper's own measured value (99.9% on an identical-prefix call) is a source-level engineering measurement, not a theoretical ceiling, and your own prefix-stability bugs will show up here first.
Hold the substrate fixed when attributing a change to the harness, the same discipline agent evaluation requires: pin the model, the price table, and the task set, and vary only the orchestration code, exactly as this paper's paired-swap design does.

How to maintain the gains¶

Treat prefix stability as a correctness invariant, not a nice-to-have. The paper enforces the byte-stable-prefix rule structurally (the cache-marker logic refuses a breakpoint at or after the first volatile message) precisely because a single silent regression (a reordered JSON key, a re-inserted timestamp) reintroduces full-price billing on the dominant cost term; see the matching failure mode in agentic loop economics.
Re-run the capability-floor check after every model swap or upgrade. A weaker model added to your fleet may not clear the reliability threshold your stronger models cleared for a delegated or orchestration-heavy feature; the paper's own regression pattern (all seven quality regressions concentrated on the three smaller models) is the shape to watch for.
Re-price after every provider list-price or cache-discount change. The dollar figures in this paper are price-table-dependent (Section 8, "Price-table dependence"); the token and latency deltas are not, so keep both dashboards and do not let a stale price table quietly overstate or understate the savings.

Results¶

Blended across all six models and 22 tasks (Table 3 of the paper):

Dimension	Baseline	Harness	Delta	Reading
Quality (task-completion)	0.78	0.81	+0.03	wash at n=22
Cost / task	$0.21	$0.12	-41%	reported; uncertainty not published
Wall-clock / task (median)	48s	27s	-44%	reported; uncertainty not published
Tokens / task	14.2k	8.8k	-38%	reported; uncertainty not published
Quality per dollar	3.71	6.75	+82%	derived
Completions per million tokens	54.9	92.0	+68%	derived

Per-model cost and latency (Table 4):

Model	Cost base	Cost harness	Delta	Latency base	Latency harness	Delta
Claude Sonnet 4.6	$0.24	$0.15	-39%	52s	31s	-41%
Gemini 3.1	$0.19	$0.13	-33%	49s	29s	-40%
Gemini Flash 3.5	$0.18	$0.07	-61%	60s	27s	-55%
Qwen 3.6	$0.16	$0.09	-44%	44s	29s	-33%
GLM 5.1	$0.21	$0.11	-47%	47s	29s	-38%
Palmyra X6	$0.25	$0.12	-52%	50s	26s	-48%

Harness leverage: per-model mean quality gain across eight scored capabilities correlates with baseline capability at r=0.99 (n=6, suggestive not conclusive): Palmyra X6 +0.079, Sonnet 4.6 +0.073, Gemini 3.1 +0.050, GLM 5.1 +0.028, Flash 3.5 +0.010, Qwen 3.6 -0.031. The one net-new capability, sub-agent delegation (no baseline arm exists for comparison), scored 0.86 (Palmyra X6), 0.85 (Sonnet 4.6), 0.70 (Gemini 3.1), 0.58 (GLM 5.1), 0.45 (Flash 3.5), 0.42 (Qwen 3.6): a usable reliability threshold only on the two strongest models.

Mechanism-family comparison across six widely used agent systems, assessed from public documentation (Table 1; explicitly not head-to-head measurement):

System	Deployment class	Model-portable	Cache policy	Compaction	Firewalled delegation	Zero-token waits	Per-task accounting
Claude Code	single-user client	no	yes	yes	partial	no	no
Claude Cowork	single-user client	no	yes	yes	partial	no	no
LangGraph	library	yes	app	app	app	partial	no
CrewAI	framework	yes	app	partial	no	no	no
AutoGen / AG2	framework	yes	app	app	no	partial	no
Hermes Agent	personal agent	yes	partial	yes	yes	partial	no
Writer Harness	multi-tenant runtime	yes	yes	yes	yes	yes	yes

("app" means the framework leaves that concern to the application to build and budget.)

Failure modes¶

Silently reconciling incompatible precision. The paper reports -41%, while its rounded $0.21 and $0.12 values reproduce -42.9%. The published values do not establish why. Report both and mark the unresolved precision gap.
Exposing an orchestration-heavy feature to a model below its capability floor. Sub-agent delegation and other orchestration-heavy capabilities (MCP tool use, Playbooks) are exactly where all seven quality regressions concentrated, all on the three smaller models in the panel.
Treating quality parity as a quality win. The paper calls +0.03 at n=22 a wash. The reported efficiency deltas are consistent in direction in the panel, but the paper does not publish enough uncertainty information to independently establish significance.
Reading Table 1 as a benchmark. It is a design-time comparison from public documentation and a source-level study conducted during this harness's own design, not a controlled cross-harness measurement; treat any specific vendor's row as informative, not authoritative.
Ignoring the single-vendor, single-pair, single-run-baseline threats to validity the authors themselves list: the baseline was run once and frozen, so its run-to-run variance is unmeasured; the harness-leverage correlation rests on six models; and the entire comparison is one vendor's harness against one vendor's own baseline.

References¶

Sayed Ali et al. (Writer, Inc.), The Harness Effect: How Orchestration Design Sets the Token Economics of Enterprise Agentic AI: https://arxiv.org/abs/2607.06906
Gu, From Model Scaling to System Scaling: Scaling the Harness in Agentic AI: https://arxiv.org/abs/2605.26112
Yao et al., Harness-Bench: Measuring Harness Effects Across Models in Realistic Agent Workflows: https://arxiv.org/abs/2605.27922
Anthropic, How We Built Our Multi-Agent Research System: https://www.anthropic.com/engineering/multi-agent-research-system
Anthropic, Prompt Caching (Claude platform documentation): https://platform.claude.com/docs/en/build-with-claude/prompt-caching
Liu et al., Lost in the Middle: How Language Models Use Long Contexts: https://arxiv.org/abs/2307.03172