Markdown

Harness Handbook: behavior-first code localization¶

Scope: generating and maintaining a source-grounded map that routes a coding agent from a requested behavior to the stages, state registers, files, functions, and tests that implement it. The page covers Harness Handbook at audited repository commit dc222e64d2a2bddb5b293ba7b93d4a54aeb32e38, its Behavior-Guided Progressive Disclosure (BGPD) workflow, the public implementation, and the limits of the paper's planning-only evaluation. For the runtime that executes the resulting change, start with harness architecture.

The shell commands are reference templates. The dependency versions shown below were installed and the Python-only static-analysis phase was executed against the generator's own source. The LLM-assisted generation phases, planner, patch execution, and resynchronization path were not executed. The paper's 60-case benchmark cannot be rerun from the public repository because the authors removed its evaluation artifacts.²

What it is¶

Harness Handbook is a behavior-oriented, three-level view of a codebase:¹

L1 system overview: execution model, stages, and global data flow.
L2 stage pages: responsibilities, inputs, outputs, dependencies, and local state.
L3 source-backed leaves: functions or contiguous regions for a small codebase, and files for a large codebase.
State-register view: shared state that couples otherwise separate stages.

The representation complements a file tree and call graph. A request such as "change how tool environment variables are validated" may touch parsing, state propagation, tool launch, and tests even when those locations share no useful identifier. The handbook names the behavior, exposes the shared state, and retains locators into the current source.

Two construction modes trade setup effort for granularity:

Mode	Leaf	Behavioral skeleton	Appropriate use
Small codebase	Function or contiguous region	Supplied by the maintainer	A compact harness whose execution stages are known and stable
Large codebase	File	Inferred during synthesis	A repository where complete file coverage matters more than symbol-level precision

The handbook is derived data, not an authority. Every proposed edit must be re-opened in the current source, and an entry whose locator or fingerprint no longer resolves must be rejected or regenerated.

Why use it¶

A repository map answers "what code exists." Behavior localization answers "which current locations jointly implement this requested change." That distinction matters in an agent harness because behavior crosses control flow, state, tool boundaries, fallback paths, and mirrored tests.

The paper evaluates 30 authored requests on Terminus-2 and 30 on Codex. A read-only DeepSeek-V4-Pro planner received the handbook in the assisted arm. LLM-judge preference for the handbook plan exceeded the baseline by 10.0 percentage points on Codex and 18.9 points on Terminus-2, while planner-consumed tokens fell from 102k to 89k and 58k to 53k respectively.¹ Agreement with two model-generated reference plans improved in all reported recall, precision, and F1 comparisons. These are planning and model-agreement results, not patch success rates.

The paper's macro F1 against its two model-generated reference plans is:

Harness / reference plan	File F1, baseline to handbook	Symbol F1, baseline to handbook
Codex / Opus 4.8	46.6 to 61.8	38.3 to 57.1
Codex / GPT-5.5	47.3 to 52.3	43.8 to 51.2
Terminus-2 / Opus 4.8	74.1 to 84.7	64.8 to 77.1
Terminus-2 / GPT-5.5	76.5 to 89.3	73.0 to 89.3

The values measure overlap with model-authored plans, not human-validated localization correctness. The paper excludes missing outputs and planner errors from this table without reporting their counts.

The evidence supports a narrower operational hypothesis: precomputed behavioral structure can improve edit localization after its construction cost is amortized. It does not establish successful patches, lower end-to-end compute, human comprehension, or parity with stronger coding models. The target revisions, construction cost, planner-error counts, confidence intervals, human gold labels, and paper-matching benchmark artifacts are unavailable.¹²

When to use it (and when not)¶

Use it when behavioral changes routinely cross files or stages, repository search misses shared-state coupling, and the same codebase receives enough work to amortize handbook generation.
Use function leaves when maintainers can supply a reliable execution skeleton and source regions remain tractable.
Use file leaves when the repository is large, file coverage is the primary invariant, or symbol extraction varies across languages.
Regenerate instead of resynchronizing after a stage-boundary change, major file reorganization, parser upgrade, or language migration invalidates the hierarchy.
Do not use it as the source of truth. A fresh source read, tests, and review remain mandatory before an edit.
Do not infer execution quality from the paper. Its experiment stops at read-only localization and planning; it does not apply patches or run tests.
Do not deploy the current generator unpinned. The repository has no dependency lock, package manifest, release, or license file at the audited commit.²

Architecture¶

Construction separates deterministic program facts from model-generated organization. BGPD then reads coarse-to-fine and returns to source before planning.

flowchart LR
  SRC["Current source tree"] --> STATIC["Static adapters<br/>files, symbols, calls, state access"]
  STATIC --> GRAPH["Program graph and unresolved-edge log"]
  GRAPH --> ORG["Behavioral organization<br/>human skeleton or LLM-inferred stages"]
  ORG --> DOC["L1 overview, L2 stages, L3 leaves"]
  GRAPH --> REG["Cross-stage state registers"]
  DOC --> ROUTE["BGPD coarse-to-fine routing"]
  REG --> ROUTE
  ROUTE --> EXPAND["Expand through calls and registers"]
  EXPAND --> VERIFY["Re-open current source and verify locators"]
  VERIFY --> PLAN["Source-grounded edit plan"]
  SRC --> VERIFY
  DIFF["Accepted source change"] --> SYNC["Incremental resynchronization or rebuild"]
  SYNC --> DOC

Static adapters collect functions, signatures, source ranges, internal calls, named external boundaries, and observed state access. Unresolved calls are recorded rather than invented. LLM-assisted phases classify or infer stages and synthesize descriptions. This separation lets operators audit the deterministic graph independently from the narrative layer.

BGPD routes through six checks:

Select candidate L1 and L2 stages from the request.
Follow state-register links into coupled stages.
Select L3 functions, regions, or files.
Expand candidates along retained call edges.
Re-open each locator in the current repository.
Keep only source-backed evidence for the edit plan.

How to use it¶

Pin the repository and dependencies before the first scan. This setup reproduces the environment used for the local audit; it is not an upstream lock file.

git clone https://github.com/Ruhan-Wang/Harness_Handbook.git
cd Harness_Handbook
git checkout dc222e64d2a2bddb5b293ba7b93d4a54aeb32e38
python3 -m venv .venv
. .venv/bin/activate
pip install tree-sitter==0.25.2 tree-sitter-language-pack==0.13.0 \
  pyyaml==6.0.2 requests==2.32.4 markdown==3.8.2 pygments==2.19.2

Run the no-LLM static phase first. The audited Python-only command completed against handbook_generate_large itself:

python handbook_generate_large/run.py \
  --lang python \
  --source-root handbook_generate_large \
  --work-dir /tmp/handbook-python \
  --phase 1

Observed terminal summary:

[scan] 37 files
[build] functions=435 kept=1060 dropped=2743
[build] internal=445 boundary=93
[build] boundary=280 boundary_constructor=32 internal_constructor=132 internal_func=413 param_method=107 self_method=96

The resulting graph contained 538 nodes and 1,060 edges. The same command with --lang auto scanned one Shell file after the Python files, then failed with AttributeError: 'tree_sitter.Node' object has no attribute 'kind' in adapters/base.py. Python-only analysis is therefore the validated boundary for this dependency set; the advertised automatic multi-language path is not validated by this run.

For a small codebase, supply a reviewed YAML stage skeleton and use handbook_generate_small/run.py. For a large codebase, the large pipeline reads every file, synthesizes a stage skeleton, organizes files, and narrates the hierarchy. Phases after static extraction require an OpenAI-compatible endpoint. Treat their output as a draft that needs locator and source-evidence checks.

How to develop with it¶

Keep deterministic evidence and model inference in separate artifacts:

Persist path, symbol, range, call edge, state access, parser language, and source fingerprint as machine-readable facts.
Record unresolved calls and unsupported syntax explicitly. Missing graph edges are an uncertainty signal.
Require every L3 description to cite a resolvable locator and current source excerpt.
Version the skeleton, generator commit, dependency lock, model identifier, prompts, and source revision together.
Test routing with search-hostile changes whose related sites share state or behavior but not names.

The following stdlib model validates the core routing boundary. It begins with one stage seed, expands through calls and a shared register, rejects a stale locator, excludes unrelated code, and verifies that a newly stale source fingerprint is removed. It is a minimal invariant test, not the project's generator. Run: python3 bgpd_locator.py.

# bgpd_locator.py - runnable with the Python standard library.
from dataclasses import dataclass

@dataclass(frozen=True)
class Entry:
    name: str
    stage: str
    registers: frozenset[str]
    calls: frozenset[str]
    fingerprint: str

entries = (
    Entry("schema", "input", frozenset({"command_env"}), frozenset({"forward"}), "v1"),
    Entry("forward", "dispatch", frozenset({"command_env"}), frozenset({"spawn"}), "v1"),
    Entry("spawn", "execution", frozenset({"command_env"}), frozenset(), "v1"),
    Entry("test_tool", "verification", frozenset({"command_env"}), frozenset(), "v1"),
    Entry("stale_doc", "docs", frozenset({"command_env"}), frozenset(), "old"),
    Entry("telemetry", "observability", frozenset({"metrics"}), frozenset(), "v1"),
)
current = {entry.name: "v1" for entry in entries}
current["stale_doc"] = "v2"

def locate(stage, current_fingerprints):
    selected = {entry.name for entry in entries if entry.stage == stage}
    if not selected:
        raise ValueError("request did not resolve to a stage")
    changed = True
    while changed:
        changed = False
        registers = set().union(*(entry.registers for entry in entries if entry.name in selected))
        calls = set().union(*(entry.calls for entry in entries if entry.name in selected))
        for entry in entries:
            if entry.name in calls or entry.registers & registers:
                if entry.name not in selected:
                    selected.add(entry.name)
                    changed = True
    verified = sorted(
        entry.name for entry in entries
        if entry.name in selected and current_fingerprints.get(entry.name) == entry.fingerprint
    )
    stale = sorted(entry.name for entry in entries if entry.name in selected and entry.name not in verified)
    return verified, stale

verified, stale = locate("input", current)
expected = ["forward", "schema", "spawn", "test_tool"]
assert verified == expected
assert stale == ["stale_doc"] and "telemetry" not in verified

changed_source = dict(current)
changed_source["spawn"] = "v2"
assert "spawn" not in locate("input", changed_source)[0]
try:
    locate("missing", current)
    raise AssertionError("unroutable request was accepted")
except ValueError:
    pass

print("verified sites:", verified)
print("stale sites rejected:", stale)
print("coverage: keyword=1/4 behavior-map=4/4")

Executed output:

verified sites: ['forward', 'schema', 'spawn', 'test_tool']
stale sites rejected: ['stale_doc']
coverage: keyword=1/4 behavior-map=4/4

How to maintain it¶

Resynchronize only after an accepted source change, not after a planner proposal. The public helper expects an edited source tree, a plan.md, and optionally a diff. Function mode uses line-independent fingerprints; file mode detects file-set and hash changes. A hierarchy change should force full regeneration.

Maintenance gates should reject publication when:

a locator no longer resolves or its source fingerprint changed;
a changed file has no L3 owner;
a new state register crosses stages without an index entry;
an adapter's unresolved-edge count rises unexpectedly;
a stage gains overlapping or contradictory ownership;
the generated handbook revision does not match the repository revision.

Keep the old handbook until the new artifact passes those gates. A stale map that looks plausible is more dangerous than no map because it turns outdated narrative into confident edit guidance.

How to run it in production¶

Treat generation as an offline, versioned documentation build. Do not put LLM synthesis or handbook repair on the request path of a coding agent.

Immutable inputs: archive the exact source revision, generator revision, dependency lock, skeleton, prompts, and model configuration.
Least privilege: generation needs read-only source access. The planner is read-only in the released repository. Patch execution belongs to a separate reviewed workflow.
Cost accounting: measure static-analysis time, LLM construction tokens, handbook size, planner tokens, source reads, and regeneration frequency. The paper reports planner tokens only and excludes construction cost.
Quality evaluation: score file and symbol localization against a human-reviewed change manifest, then compile and test the resulting patch. Model-generated reference plans are useful secondary labels, not ground truth.
Freshness SLO: fail closed when source and handbook revisions diverge. Rebuilding on merge is safer than silently using stale locations.
Language canary: run a tiny repository through every enabled adapter after dependency upgrades. The audited --lang auto failure shows why Python success cannot stand in for multi-language validation.

Failure modes¶

Stale locators. The narrative remains readable after code moves, but the cited source no longer matches. Re-open and fingerprint every candidate before planning.
Missing cross-language edges. Dynamic dispatch, macros, generated code, shell wrappers, and foreign-function boundaries can disappear from a static graph. Log unresolved edges and supplement them with tests and runtime traces.
Automatic-language parser crash. At the audited commit and dependency versions, the Shell scan calls a missing Tree-sitter node attribute. Pin adapters independently and canary each language before enabling --lang auto.
LLM organization error. A model can put a file in the wrong stage or invent a responsibility. Require source evidence and human review for the skeleton and register map.
False confidence from coverage. File-as-leaf mode can cover every file while still describing the wrong behavior or missing a symbol within a file.
Evaluation leakage. DeepSeek-V4-Pro is both the planner and one of three judges; all reference plans are model-generated, and missing outputs or planner errors are excluded without counts.
Paper inconsistency. Figure 5 reports a 3.3-point Terminus-2 Query gain, while Section 4.2.3 says all six harness-by-type gains fall between 16.3 and 33.3 points. Use the figure-derived 3.3 to 33.3 range.¹
Irreproducible benchmark. The released repository omits the paper's requests, target revisions, reference plans, result files, grading code, and failure counts.²
Uncounted build cost. Lower planner token use can coexist with higher end-to-end cost when handbook construction and resynchronization are included.
Planning mistaken for correctness. Better localization does not prove that an edit compiles, passes tests, preserves safety properties, or satisfies the request.

References¶

Harness Handbook project and hosted examples: https://ruhan-wang.github.io/Harness-Handbook/
Wang et al., Harness Handbook: Making Evolving Agent Harnesses Readable, Navigable, and Editable: https://arxiv.org/abs/2607.13285
Harness Handbook source repository: https://github.com/Ruhan-Wang/Harness_Handbook
Audited source revision: https://github.com/Ruhan-Wang/Harness_Handbook/commit/dc222e64d2a2bddb5b293ba7b93d4a54aeb32e38
NexAU, the agent framework used by the released planner helper: https://github.com/nex-agi/NexAU

Wang et al., arXiv 2607.13285 v1, submitted 14 July 2026. The experiment covers 60 authored requests across Terminus-2 and Codex, evaluates read-only planning, and uses LLM judges plus model-generated reference plans. It does not execute or test patches. Figure 5 conflicts with Section 4.2.3 on the lower bound of type-specific gains. https://arxiv.org/abs/2607.13285 ↩↩↩↩
Harness Handbook repository at audited commit dc222e64d2a2bddb5b293ba7b93d4a54aeb32e38. The repository contains large/file and small/function generators, planner packaging, and resynchronization code, but explicitly omits evaluation, benchmark, grading, executor, generated handbooks, and run artifacts. It has no dependency lock, release, package manifest, or license file at that revision. https://github.com/Ruhan-Wang/Harness_Handbook/commit/dc222e64d2a2bddb5b293ba7b93d4a54aeb32e38 ↩↩↩↩