Markdown

Workspace-Bench: agent file-dependency benchmark¶

Scope: Workspace-Bench 1.0 (arXiv 2605.03596), a benchmark for "workspace learning," whether an agent can identify, reason over, and exploit the dependencies among the many heterogeneous files in a real worker's workspace, rather than answer over a handful of pre-specified files. This page is the benchmark deep dive: what a task and its file dependency graph are, how the 28 harness-and-model configurations were scored with Agent-as-a-Judge, what they actually scored, and the validated metric model behind the numbers. It is a workspace-specific instance of general agent evaluation, a sibling of the ops-domain AIOpsLab and the security-domain cybersecurity agent evaluation, and it depends on the harness architecture and context and memory an agent brings to a large file tree.

The paper has been revised since its initial submission: v1 (May 2026) evaluated a roster that included Anthropic's Opus-4.7 and a "Claude Code" harness, with a since-deleted "frontier comparison" subsection; the current version (v4, cited below) dropped both, replacing them with different models and harnesses, which changed nearly every headline number. This page cites the current v4 text; verify against the live arXiv page before quoting a number, since the paper has moved once already. Even within v4 the paper is not fully internally reconciled: its abstract (best "~60%", mean 43.3%, matching a direct computation over Table 4's 28 rows) disagrees with its own Section 5.2 body text ("approximately 27% to 60%," mean "45.1%"), and its Introduction and Section 6 still quote the difficulty-decline figures from the superseded v1 roster (57.6% Easy to 40.5% Hard) rather than the current Section 5.3/Figure 7 figures. This page uses the abstract/Table-4-computed figures as authoritative wherever the paper's own sections disagree, and flags each discrepancy below. Model and harness names are 2026-era and will date further; verify current results on the repository. The Python scoring model is executed and asserted (stdlib only).

flowchart TB
  subgraph BUILD["Benchmark (human-curated)"]
    WS["5 worker profiles<br/>20,476 files, 74 types, up to 20GB"] --> TASK["388 tasks (Lite: 100)<br/>each with a file dependency graph<br/>7,399 rubrics total"]
  end
  TASK --> RUN["Agent under test<br/>4 harnesses x 7 models = 28 configs"]
  RUN -->|"reads workspace, produces output files + trajectory"| OUT["Output files"]
  OUT --> JUDGE["Agent-as-a-Judge (Seed-2.0-Lite)<br/>rubrics + inputs + trajectory"]
  JUDGE --> M["Metrics:<br/>Rubric Pass Rate, TCR@p,<br/>Node F1 / Edge F1"]
  M --> RESULT["Best 61.0% (DeepAgent+GLM-5.1)<br/>mean 43.3% | human 80.7%"]

What it is¶

Workspace-Bench evaluates an agent dropped into a realistic professional workspace: a directory tree of thousands of files across dozens of formats, where solving a task means first finding which files matter and how they relate, then acting on them. The paper defines the skill as workspace learning: agents must "identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace."¹ The argument for a new benchmark is that prior work "largely evaluate[s] agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored," typically with monolithic single-style file systems and fewer than ten file modalities.¹

The construction is human-curated, not LLM-generated. The authors analyzed 154 authentic task scenarios sourced from the Lark platform at ByteDance, then had 25 annotators aligned to five worker roles write the tasks: each writes a natural-language instruction, identifies the required input files, produces a reference output, and designs the rubrics, at an average of over three hours of expert annotation per task.³ The tasks are intentionally under-specified, so the agent must inspect the workspace and recover the dependencies rather than being handed them. Crucially, every task carries a human-annotated file dependency graph: "the minimal set of essential file paths that an agent must access or use to solve the task correctly," with each directed link between files counted as one edge.³

The benchmark ships at two sizes: the full set of 388 tasks and a 100-task Lite subset (about 70% cheaper to run) that preserves the distribution and is what the experiments actually use.²

Why use it¶

It measures dependency reasoning, not just file finding. The scoring separates whether an agent found the right files (Node F1 over the dependency graph) from whether it understood how they relate (Edge F1), and the paper's central finding is that these come apart: "Node F1 scores are significantly higher than the Edge F1 scores, indicating that comprehending the relationships between files is inherently more challenging for agents than merely identifying task-relevant files."⁶
It is realistically large and heterogeneous. Workspaces average 4,095 files (the Researcher workspace peaks at 11,020 files across 2,059 directories), spanning 74 file types (spreadsheets 37.5% of files, documents 35.3%, code and config 12.7%), with structural noise (redundant folders, ambiguous names, archive directories) deliberately injected so navigation itself is part of the task.²
It exposes where agents break down. Performance declines monotonically with difficulty (the paper's current Section 5.3/Figure 7 figures are 51.4% Easy, 46.0% Medium, 35.7% Hard; its own Introduction and Section 6 still quote an older 57.6%/40.5% Easy/Hard split left over from an earlier version, an internal inconsistency worth knowing about rather than a KB error), and two dimensions, heterogeneous file understanding and lineage tracing, "consistently rank at the bottom across most agents," which tells a platform team which capabilities to stress-test before trusting an agent on a real file tree.⁶
It quantifies the human gap. Twenty domain experts (allowed to use agents as assistive tools) reach 80.7% and, unlike the agents, do not degrade as tasks get harder, so the benchmark gives a concrete target and shows current agents are far from it.⁵

When to use it (and when not)¶

Use it to compare agent harnesses and base models on file-dependency reasoning before deploying an agent against a real document or code workspace, the same pre-deployment gate this KB recommends for agent evaluation generally. The 28-configuration grid (4 harnesses times 7 models) is designed for exactly this cross-comparison.
Use it to separate the harness's contribution from the base model's: the paper finds that on Easy tasks the base model dominates and the harness barely matters, while from the "file-to-file dependency reasoning" stage onward "the harness contributes more consistently to task execution than the underlying foundation model."⁷
Use the Lite subset for routine regression runs; it preserves the distribution at about 30% of the cost, and the published experiments are all on Lite.²
Do not read the scores as a fixed capability ceiling. They are 2026-era models and harnesses at one point in time; the ranking is informative, the absolute numbers will move.
Do not assume transfer to your file tree. The workspaces are five internet-company personas built from public resources; a domain with different formats, naming conventions, or dependency structures needs its own tasks before a Workspace-Bench score means anything for it.

Architecture¶

Every task pairs a natural-language instruction with a workspace and a file dependency graph, and is graded against a set of rubrics (fine-grained binary propositions, averaging 19.1 per task, grouped into foundational, procedural, and result-oriented criteria). An auxiliary step converts vague criteria such as "Is the calculation correct?" into data-grounded assertions such as "Does the final value equal a specific value?", and experts cross-validate all tasks, graphs, outputs, and rubrics for objectivity.³

The benchmark spans five worker profiles (Operations Manager, 122 tasks; Logistics Manager, 115; Researcher, 67; Backend Developer, 43, which alone carries 43 distinct file extensions; AI Product Manager, 41) and labels each task along six workspace-learning dimensions (multi-label, so counts do not partition the set): Workspace Exploration (67.5% of tasks), Task-Supporting Files Utilization (61.3%), Result-Providing Files Utilization (54.4%), Content Relations Understanding (43.8%), Semantic Heterogeneous File Understanding (36.1%), and Lineage Tracing (35.1%). Difficulty is Easy 14%, Medium 53%, Hard 33%, and separately by dependency-edge density (low 0-2 edges 33.8%, moderate 3-5 edges 36.9%, high 6 or more 29.4%).²

The evaluation harness runs workspaces in isolated sandboxes with dual parallelism (five profiles concurrent plus task-level replicas), captures output files through multi-strategy extraction, and restores each workspace with a parallel-BFS rollback that diffs the manipulated tree against a baseline snapshot. Scoring is Agent-as-a-Judge: a judge model (Seed-2.0-Lite, held constant across all configurations) is given the deduplicated output files, the original inputs, the task rubrics, and the evaluated agent's execution trajectory, and emits per-rubric binary correctness with confidence and categorized error types.⁴

How to use it¶

Three metrics carry the results.⁴ Rubric Pass Rate is the headline: passed rubrics over total rubrics. TCR@p (Task Completion Rate at threshold p) is the fraction of tasks that pass at least p% of their rubrics, reported at p in {30, 50, 70, 90, 100}. Dependency Graph Recognition is the Node F1 and Edge F1 of the agent's predicted dependency graph against the ground truth.

The evaluated set is 4 harnesses (OpenClaw, Codex, DeepAgent, and Hermes) times 7 foundation models (GLM-5.1, Qwen-3.6-Plus, MiniMax-M2.7, Kimi-2.5, GPT-5.4, Grok-4.3, and Gemini-3.1-Pro), for 28 configurations run on Workspace-Bench-Lite.⁵ The best is DeepAgent + GLM-5.1 at 61.0% (Easy 59.2, Medium 61.9, Hard 60.5, Pass@30 84.0, Pass@100 16.0), the worst is DeepAgent + Grok-4.3 at 13.8% (Easy 24.5, Medium 13.6, Hard 9.7), and the mean across all 28 (computed directly from Table 4) is 43.3%, against the human 80.7%.⁵ Note the paper's own abstract-versus-body discrepancy: the abstract states the best reaches "about 60%" with a mean of 43.3%, matching a direct recomputation from Table 4, while the body's Section 5.2 instead states a range of "27% to 60%" with a mean of "45.1%," a number that does not reconcile with either the abstract or the table beneath it; treat the abstract/Table-4-computed figures as authoritative here.⁵

How to develop with it¶

The scoring is small enough to validate directly, and doing so makes the paper's central finding concrete. This executed model implements the rubric pass rate and TCR@p (confirming TCR@p can only fall as the threshold rises, the shape of every Table 4 row), then the dependency-graph Node F1 and Edge F1, reproducing the finding that an agent can find every relevant file yet miss the relations between them:

# wsbench_metrics.py -- validated: the two scoring ideas in Workspace-Bench, the
# rubric-based task metrics and the file-dependency-graph F1, with the paper's
# central finding (Node F1 >> Edge F1) reproduced. Pure stdlib.
#
# Part A: Rubric Pass Rate and TCR@p. A task carries fine-grained binary rubrics;
# an agent's per-task score is the fraction it passes. TCR@p (Task Completion Rate
# at threshold p%) is the fraction of TASKS that pass at least p% of their rubrics.
# TCR@p is monotonically non-increasing in p (a stricter bar passes no more tasks),
# matching DeepAgent+GLM-5.1's row (the paper's current best config): Pass@30=84 >= ... >= Pass@100=16.
#
# Part B: Dependency Graph Recognition. The agent predicts the file dependency
# graph; the paper scores Node F1 (did it find the right files) and Edge F1 (did
# it get the relations between them). Finding 2: Edge F1 is universally far below
# Node F1 -- understanding inter-file relations is harder than finding the files.

from fractions import Fraction


def f1(pred: set, truth: set) -> Fraction:
    """Exact F1 over sets (no float error): 2*|P & T| / (|P| + |T|)."""
    if not pred and not truth:
        return Fraction(1)
    inter = len(pred & truth)
    denom = len(pred) + len(truth)
    return Fraction(2 * inter, denom) if denom else Fraction(0)


# ---------------- Part A: rubric pass rate + TCR@p ----------------
def rubric_pass_rate(passed: int, total: int) -> Fraction:
    assert 0 <= passed <= total and total > 0
    return Fraction(passed, total)


def tcr_at(tasks: list[tuple[int, int]], p: int) -> Fraction:
    """tasks = list of (passed_rubrics, total_rubrics). TCR@p = fraction of tasks
    whose rubric pass rate is >= p/100."""
    thresh = Fraction(p, 100)
    ok = sum(1 for passed, total in tasks if rubric_pass_rate(passed, total) >= thresh)
    return Fraction(ok, len(tasks))


# A synthetic 5-task set with a spread of rubric pass rates.
tasks = [(25, 25), (24, 25), (18, 25), (13, 25), (7, 25)]   # 100/96/72/52/28 %
# Overall rubric pass rate = total passed / total rubrics.
overall = Fraction(sum(p for p, _ in tasks), sum(t for _, t in tasks))
assert overall == Fraction(87, 125)                          # 87/125 = 69.6%

# TCR@p is monotonically NON-INCREASING as the threshold rises.
curve = [tcr_at(tasks, p) for p in (30, 50, 70, 90, 100)]
assert curve == sorted(curve, reverse=True), curve
# Concretely: >=30% -> 4/5 tasks (the 28% one fails); >=100% -> 1/5 (only the perfect one).
assert tcr_at(tasks, 30) == Fraction(4, 5)
assert tcr_at(tasks, 100) == Fraction(1, 5)
# The paper's DeepAgent+GLM-5.1 row (its current best config) is the same shape: Pass@30 (84) >= Pass@100 (16).
paper_row = {30: 84, 50: 63, 70: 45, 90: 29, 100: 16}
assert paper_row[30] >= paper_row[100]


# ---------------- Part B: Node F1 vs Edge F1 ----------------
# Ground-truth file dependency graph: which files the task depends on (nodes) and
# the directed dependencies between them (edges). "report_final depends on the
# spreadsheet, which depends on the raw export", etc.
TRUE_NODES = {"raw_export.csv", "sales.xlsx", "chart.png", "report_final.docx", "notes.md"}
TRUE_EDGES = {
    ("sales.xlsx", "raw_export.csv"),       # sheet derived from the export
    ("chart.png", "sales.xlsx"),            # chart plotted from the sheet
    ("report_final.docx", "chart.png"),     # report embeds the chart
    ("report_final.docx", "sales.xlsx"),    # report cites the sheet
}

# A capable agent finds ALL the right files (it can grep and list the workspace)
# but recovers only the obvious edges, missing the cross-format and transitive
# ones (chart<-sheet and report<-sheet) -- the paper's exact failure pattern.
pred_nodes = set(TRUE_NODES)                                 # perfect node recall
pred_edges = {("sales.xlsx", "raw_export.csv"),             # got the direct one
              ("report_final.docx", "chart.png"),           # got one embed
              ("report_final.docx", "wrong.md")}            # one spurious edge

node_f1 = f1(pred_nodes, TRUE_NODES)
edge_f1 = f1(pred_edges, TRUE_EDGES)

assert node_f1 == Fraction(1)                               # found every file
# Edge F1: 2 correct of 3 predicted and 4 true -> 2*2/(3+4) = 4/7.
assert edge_f1 == Fraction(4, 7)
# Finding 2: understanding relations (edges) is harder than finding files (nodes).
assert edge_f1 < node_f1

# Adversarial: an agent that lists every file but infers NO dependencies scores a
# perfect Node F1 and a ZERO Edge F1 -- the benchmark's whole point is that finding
# files is not the same as understanding how they depend on each other.
assert f1(TRUE_NODES, TRUE_NODES) == Fraction(1)
assert f1(set(), TRUE_EDGES) == Fraction(0)

print(f"OK Part A: overall rubric pass {float(overall)*100:.1f}%; TCR curve "
      f"{[str(c) for c in curve]} is non-increasing (TCR@30={tcr_at(tasks,30)} >= "
      f"TCR@100={tcr_at(tasks,100)}). "
      f"OK Part B: Node F1={node_f1} but Edge F1={edge_f1} (< Node F1); "
      f"files-only agent gets Node F1=1, Edge F1=0.")

Run output:

OK Part A: overall rubric pass 69.6%; TCR curve ['4/5', '4/5', '3/5', '2/5', '1/5'] is non-increasing (TCR@30=4/5 >= TCR@100=1/5). OK Part B: Node F1=1 but Edge F1=4/7 (< Node F1); files-only agent gets Node F1=1, Edge F1=0.

The Edge-F1 model is the benchmark's thesis in miniature: an agent that can list and grep a workspace scores a perfect Node F1, so file finding is close to solved, while the relations between files (which report derives from which spreadsheet, plotted from which export) are where the score collapses. When you build your own dependency tasks, weight the edges, not the nodes.

How to maintain it¶

Pin the judge and the harnesses. Scores depend on the Agent-as-a-Judge backbone (Seed-2.0-Lite here) and on each harness's version; a judge or harness change re-baselines every number, so treat the scoring stack as a versioned unit exactly as for any evaluation harness.
Report against a named task-pool revision. The full set is 388 tasks, the experiments use the 100-task Lite subset; mixing pools or adding tasks shifts averages, so state which pool and revision a score came from.
Distinguish agent failure from environment noise. Sandbox setup, file extraction, and workspace rollback can each fail on their own; log their success per episode so infrastructure flakiness does not land in the agent's accuracy column.
Reconcile the abstract and body before quoting a headline, and re-check which paper version you have. The paper's abstract (best about 60%, mean 43.3%, matching a direct computation from Table 4) disagrees with its own Section 5.2 body text ("27% to 60%," mean "45.1%"); cite the abstract/Table-4-computed figures and note the discrepancy. The paper has also been revised at least once since its initial submission, dropping an entire model and harness from the roster and deleting a results subsection, which changed the headline numbers outright; re-verify the model/harness roster and scores against the live paper before quoting either.⁵

Running it in production¶

The production role is a gate on file-dependency reasoning before an agent gets write access to a real workspace. The paper's findings give the checklist its content. First, weight the hard capabilities: performance falls monotonically with difficulty and bottoms out on heterogeneous-file understanding and lineage tracing, so test those specifically rather than trusting an averaged score.⁶ Second, do not equate spend with capability: "high interaction turns and computational costs do not necessarily guarantee superior task performance," and the paper's best configurations exceed 55% accuracy in under 30 turns while weak ones burn 40 to 60 turns in retry loops and stagnate at 30 to 45%; a token or turn budget is a real quality signal, not just a cost.⁶ Third, keep a human in the loop for anything consequential: the human-in-the-loop baseline (80.7%) beats every fully autonomous configuration across all difficulty tiers and, unlike the agents, does not degrade as tasks get harder.⁵ Match the model and harness to the domain: agents do best on Backend Developer and Researcher personas (code and structured data) and worst on the business personas, so a code workspace and a business-document workspace are not interchangeable evaluations.⁶

Failure modes¶

Missing content and reasoning errors dominate. The paper's error analysis finds Missing Content (the agent never retrieves a required file) and Reasoning Error (it retrieves but misreasons) as the majority of failures, with format and process errors marginal; an agent that looks competent on file finding can still fail on the reasoning the edges demand.⁶
Node F1 masking Edge F1. An agent that finds every relevant file posts a high Node F1 and can look strong, while its Edge F1 (the relations that actually matter) is far lower; averaging the two hides the weakness the validated model above isolates.⁶
Agent-as-a-Judge trust. Scoring depends on a judge model reading trajectories and outputs; a weaker or biased judge can miscredit, so the judge backbone is part of what must be validated, not a neutral oracle. The general evaluation-integrity cautions apply.
Persona and format overfitting. Five personas built from public resources are a benchmark, not the world; an agent tuned to these file types and naming conventions can fail on a workspace with different ones. Rotate held-out formats before trusting a headline number.
Quoting a superseded paper version. The paper's model/harness roster and headline scores have already changed once between submissions (an entire model and harness, and the results subsection built around them, were dropped); a citation or summary written against an earlier version can silently misname the best configuration. Re-check the roster and scores against the live paper, not a cached summary.
Trusting the body over the abstract by default. Within the current version, the body's own Section 5.2 states a mean (45.1%) that does not reconcile with either the abstract (43.3%) or a direct computation from Table 4 (43.3%); the usual "body is authoritative" heuristic fails here specifically because Table 4 itself agrees with the abstract, not the body prose describing it.

References¶

Tang, Zhou, Liu, Li, Wu, Wang, Huang, Zhou, Zhou, Song, Yu, Wang, Zhou, Zhou, Lv, Li, Liu, Chen, Liu, Li, Kang, and Wu, "Workspace-Bench 1.0: Benchmarking AI Agents on Workspace Tasks with Large-Scale File Dependencies" (arXiv 2605.03596): https://arxiv.org/abs/2605.03596
Workspace-Bench project page and code: https://github.com/OpenDataBox/Workspace-Bench
OfficeBench (prior multi-application office-workflow benchmark): https://github.com/zlwang-cs/OfficeBench
TheAgentCompany (prior workplace-task agent benchmark): https://github.com/TheAgentCompany/TheAgentCompany

Workspace-Bench 1.0 (arXiv 2605.03596), Abstract and Section 1. Workspace learning: agents must "identify, reason over, exploit, and update explicit and implicit dependencies among heterogeneous files in a worker's workspace." Prior benchmarks "largely evaluate agents on pre-specified or synthesized files with limited real-world dependencies, leaving workspace-level evaluation underexplored," with monolithic file systems and fewer than ten file modalities. Prior-benchmark evidence: "the best-performing AI agent achieves only 24-30% task completion in TheAgentCompany; and 47% on multi-application office workflows in OfficeBench." ↩↩
Workspace-Bench Abstract and Section 4, Table 2. Composition: 5 worker profiles, 74 file types, 20,476 files (up to 20GB), 388 tasks (Lite subset 100 tasks, about 70% cost reduction, distribution preserved), 7,399 rubrics. Per-profile task counts: Operations Manager 122, Logistics Manager 115, Researcher 67, Backend Developer 43 (43 distinct file extensions), AI Product Manager 41. Averages: 4,095 files per workspace (max 11,020, the Researcher workspace, 2,059 directories), 19.1 rubrics per task, over 3 hours of expert annotation per task, 5.1 dependency edges across 4.7 files per task. File mix: spreadsheets 37.5%, documents 35.3%, code and config 12.7%. Six dimensions (multi-label): Workspace Exploration 67.5%, Task-Supporting Files Utilization 61.3%, Result-Providing Files Utilization 54.4%, Content Relations Understanding 43.8%, Semantic Heterogeneous File Understanding 36.1%, Lineage Tracing 35.1%. Difficulty Easy 14% / Medium 53% / Hard 33%; dependency-edge density low (0-2) 33.8% / moderate (3-5) 36.9% / high (6 or more) 29.4%; rubric types result-oriented 54.8% / foundation 25.0% / process-oriented 20.2%. ↩↩↩↩
Workspace-Bench Sections 3.2-3.3. Curation is human-driven: 154 authentic task scenarios sourced from the Lark platform at ByteDance, 25 annotators aligned to the five roles writing each task's instruction, required inputs, reference output, and rubrics; tasks are intentionally under-specified. Each task has a human-annotated file dependency graph specifying "the minimal set of essential file paths that an agent must access or use to solve the task correctly" (each directed link one edge). Workspaces are built in two stages (persona-conditioned structure generation with injected structural noise, then content population by an agentic crawler over public resources plus LLM-synthesized artifacts). An auxiliary agent converts vague criteria into data-grounded assertions; experts cross-validate all tasks, graphs, outputs, and rubrics. ↩↩↩
Workspace-Bench Sections 4.5 and 5.1. Isolated sandboxes with dual parallelism (workspace-level across the 5 profiles plus task-level replicas), multi-strategy output-file extraction, and parallel-BFS workspace rollback against a baseline snapshot. Scoring is Agent-as-a-Judge: the judge (Seed-2.0-Lite, constant across configurations) receives deduplicated output files, original inputs, task rubrics, and the agent's execution trajectory, emitting binary per-rubric correctness with confidence and categorized error types. Metrics: Rubric Pass Rate (passed over total), Task Completion Rate TCR@p (fraction of tasks passing at least p% of rubrics), and Dependency Graph Recognition Rate (Node F1 and Edge F1 of predicted versus ground-truth graph). Experiments run on Workspace-Bench-Lite. ↩↩
Workspace-Bench Sections 5.2-5.3, Table 4 (current version, arXiv 2605.03596v4). 28 configurations (4 harnesses: OpenClaw, Codex, DeepAgent, Hermes; times 7 models: GLM-5.1, Qwen-3.6-Plus, MiniMax-M2.7, Kimi-2.5, GPT-5.4, Grok-4.3, Gemini-3.1-Pro). Abstract: pass rates range approximately 27% to 60%, mean 43.3% (matching a direct computation over all 28 Table 4 rows), versus human 80.7%; best DeepAgent + GLM-5.1 at 61.0% (Easy 59.2, Medium 61.9, Hard 60.5, Pass@30 84.0, Pass@100 16.0); lowest DeepAgent + Grok-4.3 at 13.8% (Easy 24.5, Medium 13.6, Hard 9.7). Discrepancy: Section 5.2's own body text states "approximately 27% to 60%" but a mean of "45.1%," which does not match either the abstract or the Table-4-computed 43.3%; treat the abstract/table figures as authoritative. Human baseline 80.7% from 20 domain experts (allowed assistive agents), stable across difficulty. Version note: an earlier submission (v1, May 2026) evaluated a different roster that included Anthropic's Opus-4.7 and a "Claude Code" harness, reporting a best of 67.7% (OpenClaw + Opus-4.7) and mean 47.4%, plus a now-deleted Section 5.4 "Comparison with Frontier Closed-Source Agent Harness" pairing "Cowork + Opus-4.7" against a 20-task challenge subset; none of that roster, those scores, or that subsection appear in the current version, which replaced Section 5.4 with "Error Analysis." Cite the current version's roster and scores, not v1's. ↩↩↩↩↩↩
Workspace-Bench Section 5.3 (findings) and 5.4 (current version, arXiv 2605.03596v4; Error Analysis was Section 5.5 in the superseded v1). (1) Monotonic decline with difficulty: Section 5.3/Figure 7 gives 51.4% Easy, 46.0% Medium, 35.7% Hard; on Easy the base model dominates, on Hard harness orchestration matters. Note the paper's own Introduction and Section 6 still quote 57.6% Easy / 40.5% Hard, an unreconciled leftover from an earlier version of the paper that has not been updated to match the current Section 5.3 figures. (2) Node F1 significantly exceeds Edge F1: "comprehending the relationships between files is inherently more challenging for agents than merely identifying task-relevant files," and heterogeneous file understanding and lineage tracing "consistently rank at the bottom across most agents." (3) Persona disparities: agents do best on Backend Developer and Researcher (code and structured data), worse on business personas. (4) Efficiency: "high interaction turns and computational costs do not necessarily guarantee superior task performance"; best configs exceed 55% accuracy in under 30 turns while weak ones use 40-60 turns and stagnate at 30-45%. (5) Human-in-the-loop (80.7%) significantly outperforms fully autonomous agents across all tiers. Error analysis: five error types, with Missing Content and Reasoning Error the majority; format and process errors marginal. ↩↩↩↩↩↩↩
Workspace-Bench Section 6. Five stages of workspace learning: L0 data-insensitive execution, L1 user-specified file execution, L2 file-to-file dependency reasoning, L3 task-to-file dependency discovery (the "Capability Singularity"), L4 workspace-native self-evolution. "From L2 onward, the harness contributes more consistently to task execution than the underlying foundation model," and the "Data Association Gap" at L3 and L4 is the bottleneck current agents "cannot yet close." There is no dedicated Limitations section in the paper. ↩