Skip to content
Markdown

Object-oriented agents (NOOA)

Scope: the agent-as-a-Python-object harness interface introduced by NVIDIA-labs OO Agents (arXiv 2607.20709, July 2026) and shipped as the nooa package. This page covers the programming model, the six model-facing capabilities the paper proposes as a rubric for any harness, the CodeAct turn and its three context regions, pass-by-reference over live Python objects, how to build and operate agents on it, and a careful read of the benchmark evidence. The generic anatomy of this layer is agent harness architecture; the tool-call interface it replaces is tools and function calling; the containment it depends on is agent sandboxing and isolation.

What is and is not reproduced here. No LLM was run: this box has no GPU, no local server, and no provider key, so every benchmark number on this page is quoted from the paper and none of it was re-measured. The Python block is executed against nooa 0.0.8 installed from PyPI on Python 3.12, and it exercises the four mechanisms that hold without a model: ellipsis dispatch, bounded previews, the in-process cell guard, and REPL namespace injection. Three of its findings contradict or qualify the paper, and they are labelled where they appear. Repository facts are read at commit 10c6846 (2026-08-07); the paper is v1, submitted 2026-07-22, and the repo has moved since.

What it is

Most agent frameworks scatter one agent across four artifacts: a prompt template, a JSON tool schema, callback code, and a workflow graph. NOOA collapses all four into a Python class and assigns each agent concern to a language feature that already exists.

Agent concept Python feature NOOA reuses
Prompt the docstring
Actions the model may take methods
Durable state typed fields on the instance
Input and output contract type annotations, enforced at runtime
Concurrency asyncio
Orchestration ordinary control flow
Failure signalling exceptions

The one new rule is the dispatch marker. A method whose body is the ellipsis literal is implemented at runtime by an LLM loop; a method with a real body stays deterministic Python. The boundary between "model decides" and "code decides" is therefore visible on one line of source.

from nooa import Agent

class SupportAgent(Agent, llm=llm):
    """You are a support agent."""          # the system prompt

    open_tickets: int = 0                    # durable, model-visible state

    def is_refund_eligible(self, days: int) -> bool:
        """True when the order is still inside the refund window."""
        return days <= 30                    # deterministic: the model calls it, never guesses it

    async def triage(self, message: str, records: list[int]) -> str:
        """Create a support ticket."""
        ...                                  # agentic: the runtime implements this

Two strategies implement the ellipsis. PredictStrategy is a single shot: render context, ask for a value, validate it against the return annotation, retry locally on failure. CodeActStrategy is the default and turns the same contract into a Python read-eval-print loop where the model calls execute_python(...) until it calls return_result(...) with a value that passes type validation. Strategies are per-method decorators, so a cheap model can serve a classification method while the agent default serves the open-ended ones.

Why use it

The paper's argument is that agent frameworks keep inventing abstractions for things Python already has, and that this costs twice: once for the developer learning a domain-specific language, and once for the model, which was trained on Python and not on the framework. Four consequences follow, and they are the reason to care.

Type annotations become executable contracts, including for termination. The model cannot end an agentic method by emitting prose. It must return a value that validates against the annotation, and a failure sends the error back into the loop. Trace analysis in the paper found this to be the sharpest behavioural difference against the comparison harnesses: OpenCode stops whenever the model replies without a tool call, and on Terminal-Bench 77% of its failed GPT-5.5 trials terminated inside ten steps. Turning "I am done" from a convention into a validated action removes a whole class of unsupported completion.

Pass-by-reference decouples data size from context size. A CodeAct method receives its arguments as live Python objects. What enters the prompt is a bounded preview: concrete type, true length, and a head and tail sample. The variable itself is a local in the execution environment, so the model can index, slice, and iterate over all of it. The amount of data an agent can process is bounded by the machine, not by the context window.

Tool output stops round-tripping through the transcript. Because results stay as live variables, the transcript does not accumulate serialized copies. On SWE-bench Verified with GPT-5.5 at xhigh effort, the paper reports NOOA reaching 82.2% with roughly 28 model calls and 1.1M tokens per task, against PI at 78.2% with 66 calls and 2.2M tokens. Half the tokens for a higher score is the claim, and it is the most operationally interesting one on this page.

The context layout is built for prefix caching. Static blocks first, then an append-only event history, then volatile dynamic blocks at the tail. Live state changing does not invalidate the cached prefix. This matters directly to serving cost; see prompt caching.

When to use it (and when not)

Reach for this interface when the work is data-heavy and typed: an agent that must process a large table, a long document, or a batch, where a JSON tool interface would force the payload through the context window. Reach for it when you want agent behaviour under the same tests, tracing, refactoring, and review as the rest of your code, and when the developers building the agent are Python developers rather than framework specialists.

Do not reach for it when the harness must run model-written code inside a boundary the agent cannot cross. In-process execution is what makes pass-by-reference work, and the paper is explicit that sandboxed code modes trade it away because they receive serialized copies at the sandbox edge. If your threat model requires containment at the code-execution boundary and you cannot put the whole agent process inside a sandbox, this design is working against you.

Do not reach for it if your fleet runs small models on the hard cases. The capability tests split cleanly here: on the stress subset the frontier group passes 93.9% and the small group 70.8%, a 23-point gap against 3.2 points on the suite overall.

Also weigh maturity. This is 0.0.x research software, released 2026-07-20, whose own changelog still reads "Initial public release" and whose README opens with a safety warning.

The six capabilities, as a rubric

The paper's more durable contribution may be the axes rather than the implementation. It names six model-facing capabilities and scores fourteen frameworks on each, and the axes work as a checklist for evaluating any harness, including one you wrote.

Capability The question it asks
Typed I/O Do agentic calls have typed arguments and a validated return value, or free text?
Pass by reference Does the model operate on live in-process objects, or on serialized copies?
Code as action Does the model act by writing code with real control flow, or by emitting one JSON call per turn?
Loop engineering Can the model itself write orchestration, or only the developer?
Object state Is durable state typed and held out of history eviction, or is the transcript the state?
Harness APIs Can the model inspect and manage context blocks and event history, or only the developer?

The bar the paper sets for a green score is that the capability is first-class in what the model sees. Developer-only versions score partial, and tracing dashboards, automatic compaction, and hidden callbacks do not count as harness APIs. On that bar the paper's Table 7 finds no other system exposing all six on one surface, and finds most systems adopting several, often behind a flag. Its own summary of the strongest recent entries (Microsoft's harness providers, Pydantic's CodeMode, OpenAI's sandbox agents, Codex's code mode) notes they shipped during the evaluation window and are mostly experimental.

Treat the table as a snapshot with a short half-life. Its scores were read from documentation and source at pinned commits retrieved 2026-07-07 to 2026-07-09, and it is scored by the authors of one of the entries. Use the axes; re-derive the cells.

Architecture

flowchart TB
  CALL["Caller: await agent.triage(message, records)<br/>arguments bound by reference"] --> RENDER

  subgraph TURN["One CodeAct turn"]
    RENDER["Render context"] --> LLM["Call the LLM"]
    LLM -->|"execute_python(code)"| EXEC["Run the cell in the session<br/>self, args and module callables are locals"]
    EXEC --> UPD["Append typed events<br/>stdout, errors, values, locals"]
    UPD --> RENDER
    LLM -->|"return_result(value)"| VAL{"Validates against<br/>the return annotation?"}
  end

  subgraph CTX["Three context regions, ordered for prefix reuse"]
    S["Static blocks<br/>system prompt, strategy instructions,<br/>execution context, doc(self)"]
    E["Event history<br/>append-only typed events"]
    D["Dynamic blocks<br/>re-evaluated every turn, at the tail"]
  end

  CTX -.->|"cached prefix survives state changes"| RENDER
  VAL -->|"no: error back to the model"| RENDER
  VAL -->|"yes"| RET["Return the value; Python resumes"]

The three regions are maintained by two programmable objects, a ContextManager for the blocks and an EventManager for the history, and both are reachable from developer code and from the agent's own generated code:

self.context["notes"] = "The user wants concise responses."      # static block
self.context.set_dynamic("todo", "self.todo.status()")           # re-evaluated per turn
recent = self.events.query(type="PythonOutput", limit=3)         # query the trace
self.events.collapse(start_tag, end_tag, summary_text="...")     # compact it

Events are typed objects with unique tags rather than lines in a flat transcript, so history stays queryable after summarization. That is the concrete form of "harness APIs" from the rubric above, and it is the axis on which the paper scores every competitor at partial or worse.

What the mechanisms actually do (runnable)

The block below is executed against nooa 0.0.8. It needs no model and no key: it probes the four mechanisms that are pure Python. Every assertion passes as written, and three of them encode a result that differs from what the paper or the repo says.

"""Probe the four NOOA mechanisms that hold without an LLM: ellipsis dispatch,
bounded previews, the cell guard, and REPL namespace injection.
Executed against nooa 0.0.8 (PyPI) on Python 3.12.
"""

import inspect

from nooa import Agent, hidden
from nooa.agentdoc import pformat
from nooa.config.truncation_config import DEFAULT_TRUNCATION_CONFIG as TC
from nooa.ellipsis_detection import has_ellipsis_body
from nooa.errors import RestrictedCodeError
from nooa.runtime.code_validator import UnifiedCodeValidator, ValidationContext
from nooa.strategies import CodeActStrategy
from nooa.unifiedllm.registry import get_llm_client

# No request is ever sent: every check below is offline. The client only has to
# construct, so the port is one nothing listens on.
LLM = get_llm_client("hosted_vllm/Qwen/Qwen3-1.7B", api_base="http://127.0.0.1:9/v1")

MAX_RETRIES = 7                      # module constant: does it reach the agent?
API_TOKEN = "s3cret-not-a-real-key"  # same question, with something worth hiding


def module_helper(x: int) -> int:
    return x + 1


def _private_helper(x: int) -> int:
    return x + 1


@hidden
def hidden_helper(x: int) -> int:
    return x + 1


class SupportAgent(Agent, llm=LLM):
    """You are a support agent."""

    open_tickets: int = 0

    def is_refund_eligible(self, days: int) -> bool:
        """True when the order is still inside the refund window."""
        return days <= 30

    async def triage(self, message: str, records: list[int]) -> str:
        """Create a support ticket."""
        ...


# 1. Ellipsis dispatch: the `...` body is the whole declaration of intent.
assert has_ellipsis_body(SupportAgent.triage) is True
assert has_ellipsis_body(SupportAgent.is_refund_eligible) is False

# 2. Bounded previews. The paper prints a 100-element list as one compact line.
records = [42, 17, 89, 33, 8] + list(range(90)) + [56, 71, 12, 45, 28]
assert len(records) == 100
paper_line = pformat(records, max_length=10, max_string=2000, max_depth=4)
assert paper_line == (
    "list(len=100, [:5]=[42, 17, 89, 33, 8], [-5:]=[56, 71, 12, 45, 28])"
), paper_line

# The shipped default for parameter rendering is max_length=25, not 10, and at
# 25 the same argument renders over 31 lines instead of one.
assert TC.prefill_format.max_length == 25
shipped = pformat(records, **TC.prefill_format.model_dump())
assert shipped.count("\n") == 30, shipped.count("\n")
assert "len=100" in shipped                 # bounded either way
assert "[:13]" in shipped and "[-12:]" in shipped

# The bound is on the preview, not the value: 1M rows still render in one line.
rows = [{"id": i, "v": i * i} for i in range(1_000_000)]
big = pformat(rows, max_length=2, max_string=80, max_depth=4)
assert big.startswith("list(len=1000000,") and big.count("\n") == 0
assert len(big) < 100 < len(rows)


def guard(cell: str, session: dict | None = None) -> bool:
    """True when the in-process cell guard rejects `cell`."""
    validator = UnifiedCodeValidator(include_repl_policy=True)
    context = ValidationContext(code=cell, exec_globals=dict(session or {}))
    try:
        validator.validate(cell, context)
        return False
    except RestrictedCodeError:
        return True


# 3a. The guard rejects every API the paper names.
for cell in (
    "eval('1+1')",
    "exec('x=1')",
    "compile('1', '<s>', 'eval')",
    "input('? ')",
    "__import__('os')",
    "globals()",
    "from math import *",
    "x = (42).__class__.__bases__",
    "import sys\nsys.exit(0)",
    "while True:\n    pass",
):
    assert guard(cell) is True, cell

# 3b. It is not a containment boundary, exactly as the README says. Three of the
# escapes the README itself names pass, and so does a plain alias of `eval`.
for cell in (
    "open('/etc/passwd').read()",
    "open('/tmp/x', 'w').write('x')",
    "import importlib.util as u\nspec = u.spec_from_file_location('m', '/tmp/m.py')",
    "obj = object()\nf = getattr(obj, 'mro', None)",
    "from pathlib import Path\nPath('/tmp/x').write_text('x')",
    "e = eval\ne('1+1')",
):
    assert guard(cell) is False, cell

# 3c. The blocking-call check resolves names against the live session, so it
# fires on a warm name and misses the same-cell import that binds it.
import time as _time  # noqa: E402

assert guard("time.sleep(5)", {"time": _time}) is True
assert guard("import time\ntime.sleep(5)") is False

# 4. Namespace injection. Module-level callables and classes reach the agent;
# constants do not, and an underscore prefix hides nothing.
namespace = CodeActStrategy()._extract_module_context(
    inspect.getmodule(SupportAgent), agent=SupportAgent()
)
assert "module_helper" in namespace
assert "_private_helper" in namespace         # the convention is not a filter
assert "hidden_helper" not in namespace       # @hidden is
assert "SupportAgent" in namespace
assert "MAX_RETRIES" not in namespace         # the paper says constants; they are dropped
assert "API_TOKEN" not in namespace

print("ellipsis dispatch      agentic=%s deterministic=%s"
      % (has_ellipsis_body(SupportAgent.triage),
         has_ellipsis_body(SupportAgent.is_refund_eligible)))
print("preview (paper, ml=10) %s" % paper_line)
print("preview (shipped, 25)  %d lines for the same 100 ints" % (shipped.count("\n") + 1))
print("preview (1M rows)      %s" % big)
print("guard rejects          eval exec compile input __import__ globals import* dunder exit while-True")
print("guard allows           open() pathlib importlib getattr  and  `e = eval; e(...)`")
print("blocking check         time.sleep warm=rejected  same-cell-import=allowed")
print("namespace injects      %s" % sorted(k for k in namespace if not k.startswith("__")))
print("all assertions passed")

Output:

ellipsis dispatch      agentic=True deterministic=False
preview (paper, ml=10) list(len=100, [:5]=[42, 17, 89, 33, 8], [-5:]=[56, 71, 12, 45, 28])
preview (shipped, 25)  31 lines for the same 100 ints
preview (1M rows)      list(len=1000000, [:1]=[{'id': 0, 'v': 0}], [-1:]=[{'id': 999999, 'v': 999998000001}])
guard rejects          eval exec compile input __import__ globals import* dunder exit while-True
guard allows           open() pathlib importlib getattr  and  `e = eval; e(...)`
blocking check         time.sleep warm=rejected  same-cell-import=allowed
namespace injects      ['Agent', 'CodeActStrategy', 'RestrictedCodeError', 'SupportAgent',
                        'UnifiedCodeValidator', 'ValidationContext', '_private_helper', '_time',
                        'get_llm_client', 'guard', 'has_ellipsis_body', 'hidden', 'inspect',
                        'module_helper', 'pformat']
all assertions passed

Four things this establishes.

The preview mechanism is real, and the paper's example is not the default. The exact string in section 3.2 reproduces at max_length=10. The shipped default for parameter rendering is prefill_format.max_length=25, and at 25 the same hundred integers occupy 31 lines. Both are bounded and both keep the full list live in the session, so the mechanism holds; the compactness in the paper does not. Budget for the default, or set the cap yourself.

The 1M-row case is the one that matters. A list of a million dicts renders in 98 characters. This is the whole argument for pass-by-reference in a single line of output, and it is the thing a file-based or JSON-based tool interface cannot do.

The cell guard stops exactly what it claims and nothing more. Every API named in section 3.4 is rejected. Every escape the README names in its own safety warning passes, which is the README being honest rather than the guard being broken. Two gaps are worth knowing anyway: rebinding e = eval and calling e(...) is not caught, because alias tracking is populated only from import statements; and the blocking-call check resolves names against the live session namespace, so a cell that imports and calls in one go slips past a check that would fire on the second cell.

Module constants never reach the agent, contrary to section 3.4. The paper says "the agent's environment (imports, methods, and constants defined in the agent's source file) are injected as locals". Constants are filtered out: injection keeps modules, classes, and callables only. This cuts both ways in practice. A config constant you expect the agent to read is invisible, and you have to make it a field or a method. Conversely, every module-level callable is injected and rendered into the prompt, and an underscore prefix does not exclude it; @hidden is the only filter.

How to use it

Install the core package and add sub-packages by name. The CLI, memory, and benchmark packages are separate distributions.

uv add nooa                     # core
uv add "nooa[cli,memory]"       # trace viewer and eval runner, long-term memory
uv add nooa-bench               # BenchAgent and the Harbor runner

Model selection goes through LiteLLM, so hosted and local endpoints use the same call:

from nooa.unifiedllm.registry import get_llm_client

llm = get_llm_client("claude-haiku-4-5")
llm = get_llm_client("hosted_vllm/Qwen/Qwen3-1.7B", api_base="http://localhost:8000/v1")
llm = get_llm_client("ollama_chat/qwen3:1.7b", api_base="http://localhost:11434")

Then write the class, mark the agentic methods with ..., and call them like any other async method. The method name, parameters, and docstring are the prompt, which has a consequence worth internalising: renaming a method changes agent behaviour. analyze_feedback and analyze_feedback_briefly produce different output from identical bodies.

Before you run anything, read what the model will see. build_prompt_data renders the whole static prefix offline, with no request sent:

data = await build_prompt_data(agent.triage, "my order is late", records)
print(data.system_prompt)   # system prompt, strategy block, execution context, doc(self)
print(data.inspect_prefill) # the pprint calls that will render your arguments

For the small agent above, that prefix measured 3,593 characters, which lines up with the paper's stated budget of roughly 1k for the NOOA system prompt plus roughly 2.5k for the CodeAct strategy instructions. It is a fixed cost on every agent, and it is cacheable.

How to develop with it

Put the boundary where the work is exact. Rules, arithmetic, parsing, and state transitions belong in real method bodies. The model then calls them instead of re-deriving them, and a bug in them is a bug you can unit-test. The ellipsis is for semantic judgment and open-ended work.

Choose the strategy per method. PredictStrategy for classification and extraction, and note that Predict renders argument values in full behind a size cap, because a single call gives the model no chance to inspect a variable. CodeActStrategy for anything iterative. Attach a small model to the Predict methods through the decorator's per-method overrides.

Let the model fan out. Inside a cell, the model can define a @strategy(PredictStrategy()) function with an ellipsis body and run it over a batch with asyncio.gather. That is subagent parallelism written in ordinary Python by the model, and it is the "loop engineering" axis in practice. It is also a cost surface: fan-out is unbounded unless you bound it.

Test agents like code. Externally initiated calls to agentic methods on one agent are serialized, and nested same-agent calls follow stack discipline with both executions appending to the same event history. Other methods and other agents run under normal async/await concurrency.

Trace by default. Every LLM call, code execution, and method invocation is traced with parent-child spans preserved. uv run nooa start-dev serves the viewer on port 5001. If the viewer is not running, tracing is silently disabled, which is convenient and also means a missing trace tells you nothing about whether the run happened.

How to run it in production

The containment boundary is the operating system, and nothing above it. This is the load-bearing operational fact on the page, and both the paper and the README say it plainly. The in-process validator exists to keep generated code from freezing the event loop and to catch common mistakes early. It is not a sandbox. The executed block above shows open(), pathlib writes, importlib, and reflection all passing the guard, exactly as the README's warning predicts. Run the agent process inside a container, a VM, or a permission system. The paper's preferred deployment is NVIDIA OpenShell.

The paper's own ARC-AGI-3 fleet is the reference configuration for a hostile workload, and it is layered rather than single-shot: a kernel-enforced per-cell sandbox with Landlock filesystem default-deny, a seccomp network block, memory and CPU caps, and a hard cell timeout, over the in-process guard, plus a per-run privilege drop. The authors report an 18-pass red-team audit of the live run finding no leakage and one escape attempt blocked by the cell guard. Note what this configuration implies: they did not rely on the in-process guard alone either.

Decide the sandbox trade before you design the agent. In-process execution is what makes pass-by-reference work. A sandboxed code mode receives serialized copies at the boundary, which is precisely the property the interface exists to avoid. If your compliance posture forces the sandbox inward, most of the token advantage on this page goes with it.

Watch the prompt surface, not just the secrets. Constants do not enter the prompt, but every module-level class and function signature does, private ones included. Keep the agent's module small, or mark helpers @hidden. Set restricted_imports and blocked_modules deliberately: the shipped blocked_modules covers network and process families such as socket, subprocess, and urllib.request, and the blocking-call list covers asyncio, importlib, multiprocessing, os, threading, and time.

MCP configuration no longer expands environment variables. The changelog records that ${VAR} placeholders in MCP server configuration are no longer resolved from the host environment; calling code must resolve secrets and pass values explicitly. If you carry an older configuration forward, it will silently stop picking up credentials.

How to maintain it

Pin the version. This is 0.0.x research software with a changelog whose only entry is the initial release, and the repo has already moved past the paper in ways that matter for reproduction. The shipped capability suite at 10c6846 declares 40 families and 116 test instances in tests/capability/config.yaml, where the paper describes 36 families and 88 instances; running the suite at HEAD across the same ten models and five runs yields 5,800 records, not the paper's 4,400. The six stress families in Table 2 are all still there, each with exactly one instance, which is what makes Table 2's 50 records per row (one instance, five runs, ten models) come out exact.

Re-read AGENTS.md in the repo before reading the source; it documents the internal conventions. Expect internal APIs to move: the probe above reaches CodeActStrategy._extract_module_context, which is private and may be renamed. Note also that a docstring example in strategies/current_call.py constructs CurrentCall(method_name=..., args=..., kwargs=...), which raises TypeError at 0.0.8 because id and decorator are required. Treat in-source examples as stale until you run them.

Results, read carefully

The evaluation is unusually broad for a framework paper, and the framing is mostly careful. Where it overstates, the paper's own tables are what show it.

Interface fluency is close to saturated, and the residue is concentrated. The capability suite passes 4,309 of 4,400 records (97.9%), with the small and efficient group at 96.0% and the frontier group at 99.2%. On the six stress families, which test batch bookkeeping, error recovery, REPL iteration, refinement, and decomposition, the aggregate falls to 254 of 300 (84.7%) and the group gap widens from 3.2 to 23 points. The failure mode also differs by scale: no frontier model scores 0 of 5 on any stress test, so every frontier failure is a reliability miss on a demonstrated capability, while 12.5% of small-model stress pairs are 0 of 5.

The reasoning-mode figures cannot be reconciled with Table 1. The text reports off-and-on pass rates of 99.5/98.6 for GPT-5.5, but Table 1 gives GPT-5.5 440 of 440, which is 100.0%; no split of 440 records into two modes averages 99.5 and 98.6 up to 100.0. Nemotron 3 Nano is worse: 52.5 rising to 84.8 in the text against 91.6% in Table 1. The same sentence names "Super-v3", which is not one of Table 1's ten models. These are evidently a different experiment, and the paper does not say so.

Two of the six stress families are half-scored by a model under test. Reading tests/capability/config.yaml, 31 of 40 families score by exact match, and seven use an LLM judge. Two of those seven are error_recovery and task_decomposition, both stress families, and both weight an LLM methodology judge at 0.5. The judge model is nemotron3-nano-30b, which is also one of the ten models being evaluated and the weakest of them at 91.6%. Half the score on two of the six hardest families is a verdict from the lineup's weakest member.

SWE-bench: the sweep is real; Terminal-Bench is narrower than it reads. On SWE-bench Verified, NOOA leads both open comparison harnesses in all five model and effort configurations tested.

Harness GPT-5.5 off GPT-5.5 high GPT-5.5 xhigh Opus 4.6 off Opus 4.6 high
NOOA 67.2 78.8 82.2 76.8 79.8
OpenCode 1.14.33 59.2 75.0 78.6 76.0 75.2
PI v0.72.1 60.8 73.6 78.2 75.6 75.8

On Terminal-Bench 2.0 the picture is mixed, and the paper reports it accurately in prose while the abstract's "the advantage is larger" framing does not survive the table. PI beats NOOA in two of five columns.

Harness GPT-5.5 off GPT-5.5 high GPT-5.5 xhigh Opus 4.6 off Opus 4.6 high
NOOA 46.1 73.0 73.0 64.0 65.2
OpenCode 1.14.33 34.8 60.7 52.8 49.4 43.8
PI v0.72.1 37.1 68.5 75.3 65.2 58.4

Two details in that table deserve attention beyond the ranking. NOOA gains nothing from xhigh over high (73.0 either way), so the most expensive setting buys nothing on this benchmark. And OpenCode drops from 60.7 to 52.8 going high to xhigh, which is a harness getting worse as the model thinks harder.

The honest summary of the effort sweep is the paper's own: margins are widest with reasoning disabled (8.0 and 6.4 points on SWE-bench, 11.3 and 9.0 on Terminal-Bench) and narrow as effort rises. The interface substitutes for planning and verification discipline the stronger configurations increasingly supply themselves. Against closed systems, 82.2% trails Codex at 88.7% and edges Claude Code at 80.8%; on Terminal-Bench the published leaderboard SOTA at submission was 84.7%, well above NOOA's 73.0%.

CyberGym compares across different network protocols. NOOA scores 86.8% on CyberGym L1 with network access blocked, which the paper presents as the top open-source result and above most closed systems. The table's own network column undercuts the ranking: the three closed entries above or near it (MDASHv2 at 95.6%, Daybreak at 85.6%, Glasswing at 83.1%) are all marked network status unknown, and the open comparator that scores 83.5% is Codex with a submission skill and an open network. The clean comparison is the one the paper does not foreground: Codex with network blocked scores 64.9%, so like for like, NOOA is ahead by 21.9 points. That is the number worth quoting.

ARC-AGI-3 is the most striking claim and the softest evidence. One NOOA agent with a 50-line world-model skill replaces a six-agent system, which is a genuine simplification result: 22 of 25 games persisted executable model code, and the memory subsystem was heavily exercised (3,262 memories written, 27,115 deliberate reads at a 99% hit rate, injection bounded at 4.1 memories per turn). The controlled ablation is the useful number, because it holds the agent fixed and swaps only the store: 50.2% RHAE with the memory subsystem against 38.4% with plain markdown files, both on GPT-5.5. That +11.8 is the cleanest evidence on the page that a typed memory store beats a directory of notes, and it is worth reading next to the filesystem as agent memory.

The headline is weaker. The 85.1% figure is a different model (GPT-5.6-sol), so the ablation is not shown to hold there. The "6.4x harness effect" against raw GPT-5.6-sol at 13.3% is footnoted by the authors themselves as indicative because evaluation budgets differ. And each configuration is one 25-game fleet, so there is no repeat and no variance estimate.

Failure modes

  • Trusting the in-process validator as a sandbox. It stops eval and friends, not open(), pathlib, importlib, or reflection, and not e = eval; e(...). Executed above. Put the process in a container.
  • Expecting a module constant to reach the agent. It does not, despite section 3.4 saying it does. Executed above. Make it a typed field or a method.
  • Assuming an underscore prefix hides a helper. Every module-level callable is injected into the REPL namespace and rendered into the prompt. Only @hidden excludes it. Executed above.
  • Budgeting context from the paper's preview example. The one-line preview is max_length=10; the shipped parameter default is 25, and a 100-element list then costs 31 lines. Executed above.
  • Renaming a method during a refactor. The name is part of the prompt. A rename is a behaviour change with no diff in the body.
  • Unbounded model-authored fan-out. The model can write asyncio.gather over generated subagent calls. Nothing in the interface caps that; you have to.
  • Reading a missing trace as a clean run. Tracing disables silently when the viewer is absent.
  • Reproducing the paper's capability numbers from HEAD. The suite has grown to 40 families and 116 instances; you will get 5,800 records, not 4,400.
  • Carrying an old MCP configuration forward. ${VAR} placeholders are no longer expanded from the host environment.
  • Choosing it for a workload that must be sandboxed at the code boundary. You lose pass-by-reference, which is most of the reason to choose it.

Open questions & validation

  • No LLM was run for this page. The token-efficiency claim (roughly half of PI's tokens for a higher score) is the one most worth reproducing yourself, because it is the claim with direct cost consequences, and it depends on a preview default this page has already shown differs from the paper's illustration.
  • The reasoning-mode inconsistency against Table 1 is unexplained. If you depend on those figures, ask the authors which record set they cover.
  • The Table 7 framework comparison is scored by an interested party at pinned July 2026 commits. Re-derive any cell you plan to act on, especially for systems whose code mode shipped during that window.
  • The +11.8 RHAE memory ablation is a single fleet on a single model. It is the most transferable result in the paper and it has n=1.
  • Version drift is the standing risk: 0.0.x, a paper at v1, and a repo that has already outrun the evaluation described in it.

References

Related: agent harness architecture | tools and function calling | the agent loop | context and memory | agent sandboxing and isolation | OpenHands agent SDK | the filesystem as agent memory | agentic context management | loop engineering | agent evaluation | prompt caching