Markdown

DevOps-Gym: benchmarking AI agents across the DevOps cycle¶

Scope: DevOps-Gym (arXiv 2601.20882, ICLR 2026), a UC Santa Barbara/NUS/Berkeley/UCLA/Google benchmark that scores AI agents on the full build, monitor, resolve, and test cycle over real Java and Go repositories, not just issue resolving in isolation: task construction (real GitHub issues plus expert-injected synthetic failures), the prefix-completion decontamination gate, the Terminal-Bench execution harness, and what four agent frameworks across five backbone LLMs actually scored. AIOpsLab covers the narrower cloud-incident-response slice of this same territory; the general harness-comparison method for coding agents is benchmarking coding harnesses; the broader evaluation discipline (rubrics, LLM-as-judge, CI gating) is evaluating agents; benchmark-integrity practices generally are in evaluation integrity and anti-gaming.

Numbers on this page are the paper's (arXiv v1, submitted 2026-01-27) and the public ucsb-mlsec/DevOps-Gym repository, confirmed public via the GitHub API on 2026-07-16 (28 stars, 3 forks, last push 2026-03-11, MIT claimed in its README though no LICENSE file is actually present at the repo root as of that check). None of the paper's benchmark scores were reproduced here; running DevOps-Gym requires Docker and the Terminal-Bench harness and was out of scope for this page. The Python block is executed and asserted: it reimplements the paper's Appendix B decontamination metrics and repository-exclusion rule from scratch and checks the result against the paper's own Table 3 numbers; it does not run DevOps-Gym or score any agent. The source paper contains several internal numeric inconsistencies (per-stage task counts, one results percentage, one framework count) that are flagged inline with exact section/table pointers rather than silently resolved.

What it is¶

DevOps-Gym is a benchmark that scores AI agents on the operational side of software engineering, not just the coding side: build and configuration, runtime monitoring, issue resolving, and test generation, plus 14 tasks that chain all four stages into one pipeline.¹ Its central argument is that prior benchmarks split cleanly into two camps that never overlap: SWE-bench-class benchmarks (issue resolving, code generation) push the "Dev" half of DevOps, while AIOps benchmarks like AIOpsLab and IT-Bench cover incident detection and root-cause analysis in simulated or narrow infrastructure settings; nothing evaluates an agent across the whole cycle on real, compiled-language repositories.¹

The benchmark targets Java and Go specifically, "large-scale enterprise SE projects that have standardized, non-trivial build systems, as well as robust monitoring infrastructure and tooling," deliberately outside the Python-dominant territory of SWE-bench and SWT-bench.² Tasks come from two sources: real GitHub issues mined from 30+ well-maintained repositories, and synthetic failures that three domain experts inject to reproduce patterns the manual analysis of 1,000+ GitHub issues (drawn from projects cited in Google's 2024 DORA report) turned up as common in practice, such as dependency-version conflicts or a memory leak from an unreleased in-memory download cache.²⁴⁵

Every real-world task is screened through a decontamination gate first: a prefix-completion probe checks whether a candidate model can reproduce a repository's own CI configs, build files, and tests verbatim, and repositories that score too "memorized" are dropped before any task is built on them.³ Repositories that survive are sanitized (git history stripped) so an agent cannot read the fix out of a future commit.²

Why it matters¶

The best evaluated setup is far from solved, on every stage. Claude Code with Claude-4-Sonnet, the top performer, scores 51.85% on build and configuration, 20.56% on monitoring, 23.87% on issue resolving, and 13.87% on test generation.⁹ Several combinations hit an outright 0.00% on monitoring: DeepSeek-V3.1 and Aider with Claude-4-Sonnet both do, despite reasonable scores on the other three stages.⁹
The cross-language cliff is large and attributable, not just "harder tasks." The exact same agent and model, OpenHands with Claude-4-Sonnet, resolves 70.4% of issues on the SWE-Bench-Verified leaderboard (Python) but only 23.87% on DevOps-Gym's Java and Go issue-resolving tasks, which the paper attributes to Python's dominance in pretraining data plus compiled-language complexity (compilation, dependency resolution, JVM/Go-specific syntax).⁶
End-to-end success is a different, much harder number than the average of the per-stage scores. Across all 14 chained pipeline tasks and every agent/model pair tested, full four-stage success is 0%, even though agents partially succeed at individual stages within a pipeline task (Table 4 lists individual cells where build, monitoring, resolving, or testing each pass alone).⁷
Test generation is consistently harder than fixing the same bug. Using the identical issue set for both tasks, every single agent/model row in Table 1 scores lower on test generation than on issue resolving, which the paper attributes to test generation needing dynamic reasoning about how a bug triggers, not just a static read of the issue description.⁶
Single-run scores understate genuine agent capability by a measurable amount. Five independent runs on a 50-task sample give Claude Code a mean of 18.80% (STD 2.40) but a Pass@5 of 26.00%; OpenHands+o4-mini's mean is 16.80% (STD 2.04) against a 20.00% Pass@5.¹⁰ The run-to-run variance is small, so a single run is not noisy, but it is a lower bound: retries surface headroom a one-shot score hides.

When to use it (and when not)¶

Use it to test whether a coding agent's DevOps competence generalizes past Python issue-resolving benchmarks, especially before trusting the same agent with Java/Go build pipelines or live monitoring tasks; the cross-language and cross-stage gaps above are the reason a SWE-bench score alone should not gate that decision.
Use it to separately probe monitoring capability (sustained tool use over time, temporal reasoning, false-positive discipline against "Normal Cases" tasks) from code-editing capability; the paper shows these do not correlate within an agent.⁵
Use the end-to-end pipeline set specifically to test context propagation across tool boundaries, since the paper's headline finding is that no agent/model pair completed one, even when they nailed individual stages.⁷
Do not port a headline number to a stack the benchmark does not cover. Everything here is Java or Go; the paper explicitly excludes CI/CD automation and infrastructure-as-code from the benchmark's scope because those stages "depend on mutable external systems (cloud APIs, Kubernetes clusters, etc.), which make both task creation and evaluation difficult."¹
Do not quote a single per-stage task count without checking the live repo. As detailed under "Interpreting and reporting scores honestly" below, the paper's own prose, its own appendix table, and the public repository's README disagree with each other on exact per-stage counts; only the headline "700+" total and the 14 end-to-end tasks (independently corroborated by Table 4's actual row count) are stable across every source.¹¹
Do not treat this as a mature, heavily-used benchmark yet. The repository was created 2026-02-01, last pushed 2026-03-11, has 28 stars, and ships no LICENSE file despite its README's MIT badge; re-verify current task counts, harness compatibility, and licensing before building a production gate around it.¹²

Architecture¶

flowchart TB
  REPOS["Candidate GitHub repos (Java/Go, active, well-maintained)"] --> GATE{"Decontamination gate:\nprefix-completion probe, exclude if C >= 0.2"}
  GATE -->|"pass"| SAN["Git history stripped (no future-commit fixes visible)"]
  SAN --> BUILD["Build & config tasks:\nCI failure-success mining (BugSwarm-style) + expert-synthesized migrations"]
  SAN --> MON["Monitoring tasks:\nexpert-injected resource/perf anomalies + Normal Cases (healthy, no anomaly)"]
  SAN --> ISSUE["Issue resolving + test generation:\nMulti-SWE-bench-style PR mining, fail-to-pass validated"]
  BUILD --> TASKDIR["Per-task dir: Dockerfile, task.yaml, setup.sh, evaluate.sh"]
  MON --> TASKDIR
  ISSUE --> TASKDIR
  TASKDIR --> TB["Terminal-Bench harness: tb run --agent ... --dataset-path tasks/<stage>"]
  TB -->|"tool calls: maven/gradle/npm, top/iostat/pprof, git/grep/sed/awk, JUnit/go test"| ENV["Containerized task environment"]
  ENV --> ORACLE["Stage oracle: build exit code + artifact check / anomaly-type match / fail-to-pass tests"]
  ORACLE --> STAGESCORE["Per-stage success rate (Table 1)"]
  STAGESCORE -->|"chained: build -> monitor -> resolve -> test, per end_to_end/ task"| PIPE{"Any stage fails?"}
  PIPE -->|"yes"| TERMINATE["Pipeline terminates: 0 for this task"]
  PIPE -->|"all four pass"| E2EOK["Full pipeline success (paper: 0/14 observed)"]

Task construction¶

Build and configuration splits into repair tasks (five failure classes: dependency version conflicts, build misconfiguration, compilation errors, tool-chain mismatches, dependency resource unavailability, mined from CI failure-success pairs following the BugSwarm methodology) and implementation tasks (build-system migration such as Maven to Gradle, target release, plugin integration, dependency upgrades, synthesized by three domain experts).⁴ Reproducing each one required rebuilding the toolchain, dependencies, and configuration by hand; roughly 40% of initially selected issues needed multiple reproduction iterations before they were reliable enough to keep.⁴

Monitoring covers six anomaly types split across two families: resource problems (memory leaks, disk leaks, handle exhaustion, CPU spikes) and performance degradations (I/O bottlenecks, inefficient SQL queries), plus tasks with no injected anomaly at all to test false-positive discipline. The 30-task set splits roughly evenly across types: memory leaks and CPU spikes at 17.1% each, disk leaks at 8.6%, and SQL inefficiencies, handle exhaustion, I/O bottlenecks, and the healthy no-fault cases each at 14.3%.⁵ Anomalies are engineered to manifest within 5-15 minutes under standard tools and were independently validated as detectable by three senior DevOps engineers using only top/free/ps-class tools, with no source-code access.⁵

Issue resolving and test generation reuse the same underlying issue set (one test-generation task per issue-resolving task) via a pipeline adapted from Multi-SWE-bench: filter merged pull requests that resolve a GitHub issue and include test modifications, then verify the associated tests actually fail before the patch and pass after it (a fail-to-pass transition).⁶ Test generation follows SWT-bench's protocol but is stricter: SWT-bench also credits partial code-coverage improvement, while DevOps-Gym only credits a generated test if it fails on the buggy code, passes on the patched code, and introduces no new failures elsewhere.⁶

End-to-end pipelines chain all four stages against one running instance: inject a build failure into an otherwise monitoring-ready repository, require the agent to fix it, then monitor the live system for a latent performance/resource issue from the monitoring pool, then patch the diagnosed root cause, then write a regression test that specifically targets the anomaly identified in stage 2. Failure at any stage terminates the task; there is no partial credit for the pipeline as a whole.⁷

Interpreting and reporting scores honestly¶

The paper is internally inconsistent about its own task counts, and the public repository does not fully agree with the paper either, so do not quote a single passage as ground truth:

Section 3.1's prose states "54 build and configuration tasks (20 synthetic tasks and 34 real-world tasks), 34 monitoring tasks (29 synthetic tasks and 5 real-world tasks), 310 issue resolving tasks, 310 test generation tasks, and 18 end-to-end tasks."¹¹ That sums to 708 four-stage tasks (726 including end-to-end).
Appendix Table 5 instead gives 48 build and configuration, 30 monitoring, 310 issue resolving, and 310 test generation tasks (698 total, no end-to-end row).¹¹
The introduction separately states the benchmark is "704 tasks... Additionally, we create 14 end-to-end pipeline tasks."¹ 704 reconciles with neither accounting: it is 4 below Section 3.1's prose total of 708 (54+34+310+310) and 6 above Appendix Table 5's total of 698 (48+30+310+310), landing closer to the prose figure than to the table.
The public repository's README states "704 real-world DevOps tasks" as its headline total, and gives a per-category table of 48/30/310/310/14, which itself sums to 712, not 704.¹²
Table 4, the actual end-to-end results table, lists exactly 14 distinct repository/anomaly rows (minio, pocketbase, spring-petclinic, tidb across their respective injected anomalies), independently corroborating the "14" figure over the "18" in Section 3.1.⁷

None of these four accountings reconcile arithmetically with each other. The only numbers stable across every source are the "700+"/"704" headline total and "14" end-to-end tasks; treat the 48/30/310/310 per-stage split (Table 5 and the README) as the best available figure and re-verify against tasks/<stage>/ directory listings in the live repo before citing an exact count in a report.¹¹¹²

Two more mismatches worth knowing before quoting the paper's own prose:

Section 1 (Introduction) states the evaluation covers "three widely used coding agents across 5 LLMs and 4 agentic frameworks." Section 4.1 ("Experimental Setup") never repeats that phrase; its own, separate sentence reads "We evaluate three best-performing agentic frameworks: OpenHands, mini-SWE-agent, and Claude Code," naming only three and omitting Aider even though Aider has its own Table 1 row. Table 1 in fact lists four framework rows (OpenHands, mini-SWE-Agent, Aider, Claude Code), matching the introduction's "4" but leaving Section 4.1's named list one short.⁹
Section 4.2's narrative claims "Claude Code consistently outperforms other agentic frameworks, reaching 58.33% success in build and configuration," but Table 1 and Figure 2b both report 51.85%/51.9% for that exact cell (Claude Code + Claude-4-Sonnet, build and configuration).⁹ Cite the table value, not the prose sentence.

How to run it against your own agent¶

The tasks ship in Terminal-Bench format and are driven by the tb CLI; this is a reference template from the repository's README, not independently executed here:¹²

# Reference template (needs Docker, Python 3.8+, uv, and Terminal-Bench). Not executed here.
git clone https://github.com/harbor-framework/terminal-bench.git
cd terminal-bench && # follow its README to install and verify: tb --help

git clone https://github.com/ucsb-mlsec/DevOps-Gym.git
cd DevOps-Gym
export LLM_API_KEY="your-api-key-here"

# Tasks are grouped under tasks/{build,monitor,issue_resolving,test_generation,end_to_end}/
uv run tb run \
  --agent <agent-name> \
  --model <model-name> \
  --dataset-path "$(pwd)/tasks/issue_resolving" \
  --n-concurrent 8

To compare against the paper's own numbers rather than an uncontrolled setup, match its evaluation protocol: temperature 0.7, top-p 0.95, max context 256K tokens, a 60-second per-call timeout with up to 3 retries, and Docker-isolated task containers.⁸ The paper evaluated three agent frameworks it explicitly names (OpenHands, mini-SWE-agent, Claude Code) plus Aider, against five backbone LLMs (Claude-4-Sonnet, o4-mini, Gemini-2.5-Pro, DeepSeek-V3.1, Qwen3-Coder-30B, the last hosted locally via vLLM), not all combinations tested (mini-SWE-Agent and Aider were only run with Claude-4-Sonnet).⁸

How to extend it with new tasks¶

The paper's own construction pipeline is the template for adding tasks, and it is explicit that this is expensive, expert-driven work, not something an agent can currently do end to end:⁴⁵

Screen the candidate repository through the decontamination gate first (see the executed model below), before investing reproduction effort in it.
For repair-style tasks (build failures, real issues): mine failure-success pairs from CI logs (build) or merged PR/issue pairs with fail-to-pass test evidence (issue resolving, following the Multi-SWE-bench pipeline).⁴⁶
For synthetic tasks (build implementation, monitoring anomalies): an expert instruments the target repository to reproduce a known failure pattern without disrupting normal behavior, then validates independently that the anomaly is both reliably triggerable and detectable through the intended tool set with no source access.⁵
Budget expert time realistically. The paper reports over 10 hours of expert SE-researcher time per monitoring or build task even with coding-agent assistance, because coding agents "tend to inject other easier-to-trigger errors" instead of reproducing the originally reported one when asked to help reconstruct an environment.²
Package the result as Dockerfile + task.yaml + setup.sh + evaluate.sh under the appropriate tasks/<stage>/ directory, matching the structure the existing 704+14 tasks use.¹²

Executed model: the decontamination gate¶

The paper's Appendix B defines contamination screening as five complementary metrics run over a 50-token prefix-completion probe (Levenshtein ratio, consecutive-prefix match length, first-mismatch position, longest common substring, sentence BLEU), a snippet is "high-risk" if any metric crosses its threshold, and a repository is excluded if its contamination rate C (fraction of high-risk snippets among 20 sampled) is at least 0.2.³ The block below reimplements all five metrics from scratch, validates an adversarial case the metric mix is specifically designed to catch, and then applies the exclusion rule to the paper's own Table 3 numbers, exposing that the paper never disambiguates which of Table 3's two reported aggregates (Max vs. Avg contamination across its two probing LLMs) is the C the exclusion rule actually gates on:

"""
Executed model of DevOps-Gym's decontamination gate (paper Appendix B, Section 3.1).
Part 1: the five prefix-completion contamination metrics and the paper's
OR-of-thresholds high-risk rule (Levenshtein ratio s>0.7, consecutive-prefix
match t>30, first-mismatch position p>30, longest common substring l>30,
sentence BLEU b>0.5). Part 2: the repository-level exclusion rule (exclude if
C >= 0.2) applied to the paper's own Table 3 numbers.
"""
from __future__ import annotations
import math
from collections import Counter


def levenshtein(a: list[str], g: list[str]) -> int:
    n, m = len(a), len(g)
    dp = list(range(m + 1))
    for i in range(1, n + 1):
        prev, dp[0] = dp[0], i
        for j in range(1, m + 1):
            cur = dp[j]
            dp[j] = prev if a[i - 1] == g[j - 1] else 1 + min(prev, dp[j - 1], dp[j])
            prev = cur
    return dp[m]


def lev_ratio(a: list[str], g: list[str]) -> float:
    if not a and not g:
        return 1.0
    return 1.0 - levenshtein(a, g) / max(len(a), len(g))


def prefix_match(a: list[str], g: list[str]) -> tuple[int, int]:
    """Consecutive matching token count t, and first-mismatch position p. Under a
    literal prefix walk these collapse to the same integer (the paper lists them as
    two separate metrics without a formula that distinguishes them)."""
    t = 0
    for x, y in zip(a, g):
        if x != y:
            break
        t += 1
    return t, t


def lcs_len(a: list[str], g: list[str]) -> int:
    """Longest common CONTIGUOUS substring, token-level (Gusfield 1997 DP)."""
    n, m = len(a), len(g)
    dp = [[0] * (m + 1) for _ in range(n + 1)]
    best = 0
    for i in range(1, n + 1):
        for j in range(1, m + 1):
            if a[i - 1] == g[j - 1]:
                dp[i][j] = dp[i - 1][j - 1] + 1
                best = max(best, dp[i][j])
    return best


def bleu(a: list[str], g: list[str], max_n: int = 4) -> float:
    """Sentence-level BLEU: n-gram precision (1..4) + brevity penalty (Papineni et al. 2002)."""
    if not a:
        return 0.0
    precisions = []
    for n in range(1, max_n + 1):
        a_ngrams = Counter(tuple(a[i:i + n]) for i in range(len(a) - n + 1)) if len(a) >= n else Counter()
        g_ngrams = Counter(tuple(g[i:i + n]) for i in range(len(g) - n + 1)) if len(g) >= n else Counter()
        overlap = sum(min(c, g_ngrams[ng]) for ng, c in a_ngrams.items())
        total = max(sum(a_ngrams.values()), 1)
        precisions.append(overlap / total if total else 0.0)
    if min(precisions) == 0:
        return 0.0
    geo_mean = math.exp(sum(math.log(p) for p in precisions) / max_n)
    bp = 1.0 if len(a) > len(g) else math.exp(1 - len(g) / max(len(a), 1))
    return bp * geo_mean


THRESH = dict(s=0.7, t=30, p=30, l=30, b=0.5)


def is_high_risk(a: list[str], g: list[str]) -> dict:
    s = lev_ratio(a, g)
    t, p = prefix_match(a, g)
    l = lcs_len(a, g)
    b = bleu(a, g)
    flagged = s > THRESH["s"] or t > THRESH["t"] or p > THRESH["p"] or l > THRESH["l"] or b > THRESH["b"]
    return dict(s=s, t=t, p=p, l=l, b=b, high_risk=flagged)


# ---- Part 1: metric behavior, happy path + adversarial paraphrase ----

# 50-token verbatim reproduction: every metric saturates.
verbatim = [f"tok{i}" for i in range(50)]
r = is_high_risk(verbatim, verbatim)
assert r["high_risk"] is True
assert r["s"] == 1.0 and r["t"] == 50 and r["l"] == 50 and r["b"] == 1.0

# Fully disjoint continuation: nothing should trip.
disjoint = [f"other{i}" for i in range(50)]
r2 = is_high_risk(verbatim, disjoint)
assert r2["high_risk"] is False

# Adversarial: change only the FIRST token, so consecutive-prefix match t/p collapse
# to 0 immediately, yet the remaining 49 tokens verbatim still form a 49-token
# common substring and BLEU stays high. A detector using only t/p would MISS this
# near-verbatim reproduction; the paper's OR-of-five-metrics rule still catches it.
paraphrased_lead = ["DIFFERENT_FIRST_TOKEN"] + verbatim[1:]
r3 = is_high_risk(paraphrased_lead, verbatim)
assert r3["t"] == 0 and r3["p"] == 0          # prefix-based metrics are blind here
assert r3["l"] >= 30 and r3["b"] > 0.5        # LCS and BLEU still catch it
assert r3["high_risk"] is True

# Boundary: thresholds are strict ">" per the paper's wording. Build a pair where
# exactly 30 tokens match at the start and the remaining 70 tokens (of 100) are
# drawn from disjoint junk vocabularies, so s, l, and b all stay comfortably below
# their thresholds too (not just t and p) while still sitting at the t/l ceiling.
g_edge = [f"tok{i}" for i in range(30)] + [f"g_junk{i}" for i in range(70)]
a_edge = [f"tok{i}" for i in range(30)] + [f"a_junk{i}" for i in range(70)]
r4 = is_high_risk(a_edge, g_edge)
assert r4["t"] == 30 and r4["p"] == 30 and r4["l"] == 30
assert r4["s"] < 0.7 and r4["b"] < 0.5
assert r4["high_risk"] is False

print("Part 1 (metric mechanics):",
      {k: (round(v, 3) if isinstance(v, float) else v) for k, v in r3.items()},
      "| boundary case:", {k: (round(v, 3) if isinstance(v, float) else v) for k, v in r4.items()})

# ---- Part 2: repository-level exclusion rule applied to the paper's own Table 3 ----
# (repo -> (Max Contamination, Avg Contamination)), transcribed from Table 3.
TABLE3 = {
    "act": (0.2000, 0.1000), "beego": (0.5625, 0.2812), "caddy": (0.2667, 0.2000),
    "checkstyle": (0.5000, 0.4583), "echo": (0.2000, 0.1000), "etcd": (0.2222, 0.1389),
    "fastjson2": (0.7333, 0.5667), "frp": (0.2500, 0.1250), "fzf": (0.1875, 0.1562),
    "gin": (0.4000, 0.2667), "go-zero": (0.1765, 0.1471), "gorm": (0.2500, 0.1562),
    "hugo": (0.1250, 0.0625), "istio": (0.3000, 0.2000), "junit-framework": (0.1000, 0.1000),
    "lazygit": (0.2308, 0.1154), "logstash": (0.2222, 0.1944), "mockito": (0.4000, 0.3000),
    "spotbugs": (0.3333, 0.2778),
}


def survives(c_value: float, threshold: float = 0.2) -> bool:
    return c_value < threshold  # "exclude repositories with C >= 0.2" (Section 3.1)


survive_on_max = {name for name, (mx, avg) in TABLE3.items() if survives(mx)}
survive_on_avg = {name for name, (mx, avg) in TABLE3.items() if survives(avg)}

# The paper defines a single scalar C ("the repository-level contamination rate") but
# Table 3 reports two: Max and Avg Contamination across the two probing LLMs (GPT-4o,
# Claude-Sonnet-4). Which one gates exclusion is never disambiguated in Section 3.1/B.
assert survive_on_max == {"fzf", "go-zero", "hugo", "junit-framework"}
assert len(survive_on_max) == 4
assert survive_on_avg == {
    "act", "echo", "etcd", "frp", "fzf", "go-zero", "gorm", "hugo",
    "junit-framework", "lazygit", "logstash",
}
assert len(survive_on_avg) == 11

# The two readings disagree on 7 of the 19 sampled repos: the choice is not cosmetic.
disagreement = survive_on_avg - survive_on_max
assert disagreement == {"act", "echo", "etcd", "frp", "gorm", "lazygit", "logstash"}
assert len(disagreement) == 7

# Boundary check on the ">= 0.2 excludes" rule itself: act and echo sit at exactly
# Max=0.2000, istio at exactly Avg=0.2000; all three must be excluded under their
# respective C, not retained by an off-by-one slip.
assert not survives(TABLE3["act"][0]) and not survives(TABLE3["istio"][1])

print("Part 2 (Table 3 exclusion):",
      f"survive on Max C: {len(survive_on_max)}/19,",
      f"survive on Avg C: {len(survive_on_avg)}/19,",
      f"disagree: {sorted(disagreement)}")

print("ALL ASSERTIONS PASSED")

Executed output:

Part 1 (metric mechanics): {'s': 0.98, 't': 0, 'p': 0, 'l': 49, 'b': 0.979, 'high_risk': True} | boundary case: {'s': 0.3, 't': 30, 'p': 30, 'l': 30, 'b': 0.289, 'high_risk': False}
Part 2 (Table 3 exclusion): survive on Max C: 4/19, survive on Avg C: 11/19, disagree: ['act', 'echo', 'etcd', 'frp', 'gorm', 'lazygit', 'logstash']
ALL ASSERTIONS PASSED

The practical lesson if you build a similar gate yourself: pick one aggregate explicitly (Max is the conservative choice, since it is the worst case across probing models) and document it, rather than reporting a single "20% threshold" the way Section 3.1's prose does, since Table 3's own sample shows most candidate repositories (15 of 19 listed) exceed 0.2 on at least one aggregate; Table 3 is a record of screened candidates, not a roster of the repos that ended up in the benchmark.³

Failure modes¶

Pitfall	Cause	Fix
Agent reports success on a migration that still fails its own test suite	Agent treats "build produces correct artifacts" as the finish line and stops after partial test failures (MobArena Maven-to-Gradle case: build succeeded, 253 tests ran, 88 failed).¹³	Score every stated success criterion independently; do not let a passing compile step stand in for a passing test suite.
Syntactically valid but functionally incomplete build config	Agent produces a config that validates (e.g. `goreleaser check` passes) but omits project-specific build tags it had no way to infer from the repo alone (Caddy's `nobadger`/`nomysql`/`nopgx` GoReleaser tags).¹³	Validate against functional output (does the produced binary actually work), not syntax-only checks.
Agent "monitors" with a single tool invocation instead of continuously	One-shot `top`/`ps` calls instead of `watch`-style continuous sampling; can coincidentally score correct on error-free instances (37% of monitoring failures).⁵	Require and reward continuous, time-windowed sampling; penalize single-snapshot answers.
Agent concludes from partial evidence after context exhaustion	Raw telemetry dumped via `cat` floods context; a session-summarization cutoff leaves other log files unchecked before the agent commits to an answer. The paper's own Appendix D.2 case study labels this "interpretation failure" (collected correct metrics, wrong or no analysis, 26% of monitoring failures), not "premature conclusion" (submitting without performing monitoring at all) -- the agent did monitor, it just stopped analyzing too early.⁵	Compact telemetry before it reaches the model context, the same lesson as AIOpsLab's behavioral findings.
False positives on healthy systems	Agent misreads normal operational variance as an anomaly; the benchmark deliberately includes Normal Cases monitoring tasks (14.3% of the 30-task pool) to catch this.⁵	Track false-positive rate on no-fault tasks as its own metric, not folded into overall accuracy.
A Python SWE-bench score is quoted as evidence of Java/Go competence	Same agent/model combo: 70.4% on SWE-Bench-Verified vs 23.87% on DevOps-Gym issue resolving.⁶	Re-measure per target language; do not transfer a Python benchmark score across the language boundary.
Per-stage average is reported as if it implied pipeline capability	0/14 full end-to-end pipeline successes across every agent/model pair, despite nonzero individual-stage scores within those same pipeline tasks.⁷	Report end-to-end success as its own, harder metric; never average per-stage rates into an implied pipeline number.
Decontamination exclusion threshold applied inconsistently	Paper defines a single scalar `C` but its own Table 3 reports two aggregates (Max, Avg) without saying which one gates the 0.2 exclusion rule; the two readings disagree on 7 of 19 sampled repos (see executed model above).³	If rebuilding this gate, pick and document one aggregate explicitly; do not assume "the 20% threshold" is unambiguous.
Quoting a single task-count figure from one passage	Section 3.1 prose (54/34/.../18), Appendix Table 5 (48/30/.../no e2e row), the intro's "704", and the repo README's own table (48/30/310/310/14=712) all disagree with each other.¹¹¹²	Pull live counts from the `tasks/<stage>/` directories in the repo; cite the source and date, not a single prose sentence.
Citing a narrative sentence over its own table	Section 4.2 claims Claude Code reaches "58.33%" on build and configuration; Table 1 and Figure 2b both report 51.85%/51.9% for that exact cell.⁹	Always cite the table value over a results-section prose summary.

References¶

Tang, Zhu, Ruan, Zhang, Yang, Li, Guo, Shi, Li, Kruegel, Vigna, Song, Wang, Wang, Ding, Liang, Guo, "DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle," ICLR 2026, arXiv:2601.20882: https://arxiv.org/abs/2601.20882 (PDF: https://arxiv.org/pdf/2601.20882, HTML: https://arxiv.org/html/2601.20882)
OpenReview submission (ICLR 2026, id bP48r4dt7Z): https://openreview.net/forum?id=bP48r4dt7Z
DevOps-Gym repository (tasks, decontamination scripts, collection pipeline): https://github.com/ucsb-mlsec/DevOps-Gym
Project website: https://www.devops-gym.com/
Terminal-Bench / Harbor (the execution harness DevOps-Gym tasks are packaged for; moved from laude-institute/terminal-bench): https://github.com/harbor-framework/terminal-bench
Jimenez, Yang, Wettig, Yao, Pei, Press, Narasimhan, "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?": https://arxiv.org/abs/2310.06770
Mündler, Müller, He, Vechev, "SWT-Bench: Testing and Validating Real-World Bug-Fixes with Code Agents": https://arxiv.org/abs/2406.12952
Zan et al., "Multi-SWE-bench: A Multilingual Benchmark for Issue Resolving": https://arxiv.org/abs/2504.02605
Chen, Shetty, Somashekar, Ma, Simmhan, Mace, Bansal, Wang, Rajmohan, "AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds," MLSys 2025: https://arxiv.org/abs/2501.06706

DevOps-Gym, arXiv:2601.20882 (2026-01-27). Abstract and Section 1: "700+ real-world tasks collected from 30+ projects in Java and Go" across build and configuration, monitoring, issue resolving, and test generation, plus end-to-end pipeline tasks; agents "struggle with issue resolving and test generation in Java and Go, and remain unable to handle new tasks such as monitoring and build and configuration." Section 2 (Related Work): positions DevOps-Gym against SWE-bench-class (Dev-side only) and AIOpsLab/IT-Bench (Ops-side, simulated or narrow) benchmarks, arguing no prior benchmark spans the full cycle on real repositories. Section 5 (Conclusion): explicitly scopes out CI/CD automation and infrastructure-as-code, "which make both task creation and evaluation difficult," as future work rather than covered stages. ↩↩↩↩
Section 3.1. Design principles (realism, agentic evaluation, complete DevOps-cycle coverage); Java/Go chosen for "standardized, non-trivial build systems" and "robust monitoring infrastructure"; manual analysis of 1,000+ GitHub issues from repositories cited in Google's 2024 DORA "State of DevOps" report to categorize task types; "even with coding agent assistance, it often exceeds 10 hours of expert work per task"; coding agents (cursor, Claude Code) observed to "inject other easier-to-trigger errors" rather than reproduce the target failure during environment reconstruction. ↩↩↩↩
Section 3.1 (contamination prevention summary) and Appendix B (full protocol). Five metrics per 50-token prefix-completion probe: normalized Levenshtein distance ratio s (threshold >0.7), consecutive matching token count t (>30), position of first token mismatch p (>30), longest common substring length l in tokens (>30, citing Gusfield 1997), sentence-level BLEU b (>0.5, citing Papineni et al. 2002); a snippet is high-risk if any metric exceeds its threshold. "The repository-level contamination rate C is defined as the proportion of high-risk snippets among the 20 samples... we exclude repositories with C >= 0.2." Table 3 reports per-repository Max Contamination and Avg Contamination columns (computed across GPT-4o and Claude-Sonnet-4 probes) without stating which aggregate is the operative C; exact Table 3 values for all 19 listed repositories are reproduced in the executed code block above. Repositories are also sanitized by removing git metadata "to prevent agents from accessing solutions through git version history," citing Kahn (2025) on SWE-bench repo-state loopholes. ↩↩↩↩
Section 3.2. Two task categories: repair (five failure types: dependency version conflicts, build misconfiguration, compilation errors, tool-chain mismatches, dependency resource unavailability; collected via a BugSwarm-style CI failure-success mining pipeline) and implementation (build-system migration, target release, plugin integration, dependency upgrades; synthesized by three domain experts). "Approximately 40% of initially selected issues required multiple iterations to achieve consistent reproducibility." Section 4.2 error-type breakdown for build/config failures (from agent run logs): toolchain/environment instrumentation limitations 33%, multi-step reasoning and sequential planning failures 23%, domain-specific knowledge gaps 37%, plus a separately reported "17% of all failures fall into the inherently difficult category." ↩↩↩↩↩
Section 3.3. Two anomaly families: resource problems (memory leaks, disk leaks, handle exhaustion, CPU spikes) and performance degradations (I/O bottlenecks, inefficient SQL queries); healthy no-fault cases included to test false positives. Anomalies "manifest within 5-15 minutes"; independently validated by three senior DevOps engineers as detectable via top/free/ps-class tools with no source access. Figure 3 anomaly distribution: Memory Leaks 17.1%, CPU Spikes 17.1%, Normal Cases 14.3%, SQL Inefficiencies 14.3%, Handle Exhaustion 14.3%, I/O Bottlenecks 14.3%, Disk Leaks 8.6%. Section 4.2 monitoring error-type breakdown: inadequate monitoring methodology (one-shot commands instead of watch-style continuous sampling) 37%, premature conclusion (submitted without completing diagnosis) 26%, insufficient temporal granularity (10-60s sampling missing transient spikes) 11%, interpretation failure (collected correct metrics, wrong or no analysis) 26%. ↩↩↩↩↩↩↩↩↩
Section 3.4. Issue resolving and test generation both derive from the same issue set via a Multi-SWE-bench-style collection pipeline (repository selection with active development, PR filtering for test-validated fixes, fail-to-pass execution validation). Test generation agents receive only the buggy repository and issue description, never the ground-truth patch, and are scored by SWT-bench's fail-to-pass protocol but without SWT-bench's partial code-coverage credit, "no tests exhibit failures on the patched code" required in addition. Section 4.2: "using the same agent and model combination (OpenHands + Claude-4-Sonnet), the resolving rate achieves 70.4% on the SWE-Bench-Verified leaderboard... yet drops dramatically to 23.87%" on DevOps-Gym; every Table 1 row scores lower on test generation than on issue resolving for the same agent/model. ↩↩↩↩↩↩
Section 3.5 and Table 4. Four-stage chained pipeline (build repair, monitoring/detection, issue resolving, validation/testing) per task instance; "failure at any point terminates the pipeline." Table 4 lists 14 distinct repository/anomaly-type rows (minio, pocketbase, spring-petclinic, tidb) each scored 0/1 per stage for Claude-4-Sonnet and GPT-5-mini. Section 4.2: "Agents achieve 0% success rate in completing all four stages on any single end-to-end task," even though individual stages (e.g. Stage 3 or 4) sometimes pass despite an incorrect Stage 2 monitoring conclusion, which the paper attributes to different performance issues sharing similar code-level manifestations. ↩↩↩↩↩
Section 4.1. Agents: OpenHands, mini-SWE-agent, Claude Code (named as the "three best-performing agentic frameworks" in the framework-description sentence) plus Aider, which has its own Table 1 row without being named in that sentence. Models: Claude-4-Sonnet, o4-mini, Gemini-2.5-Pro, DeepSeek-V3.1, Qwen3-Coder-30B (all five paired with OpenHands); mini-SWE-agent, Aider, and Claude Code each paired only with Claude-4-Sonnet. Qwen3-Coder-30B hosted locally via vLLM; all other models via official APIs. Hyperparameters: temperature 0.7, top-p 0.95, max context 256K tokens, 60-second timeout with up to 3 retries; all runs in isolated Docker containers. ↩↩
Table 1 (per-stage success rates by agent/model) and Section 4.2 narrative. Best overall: Claude Code + Claude-4-Sonnet at 51.85% (Build & Config) / 20.56% (Monitoring) / 23.87% (Issue Resolving) / 13.87% (Test Generation). DeepSeek-V3.1 and Aider+Claude-4-Sonnet both score 0.00% on Monitoring. Section 1's introduction sentence "three widely used coding agents across 5 LLMs and 4 agentic frameworks" appears on page 2, immediately before the Section 2 heading; Section 4.1's separate "Agents and Models" sentence ("We evaluate three best-performing agentic frameworks: OpenHands, mini-SWE-agent, and Claude Code") never uses the phrase "4 agentic frameworks" and names only three frameworks, omitting Aider even though Aider has its own Table 1 row and no other prose mention anywhere in the paper. Table 1's four framework rows (OpenHands, mini-SWE-Agent, Aider, Claude Code) match the introduction's "4" but exceed Section 4.1's named list of three by one. Section 4.2's prose states "Claude Code consistently outperforms other agentic frameworks, reaching 58.33% success in build and configuration," which does not match Table 1's 51.85% or Figure 2b's 51.9% for that same cell; treat the table/figure value as authoritative. ↩↩↩↩↩
Section 4.3 and Table 2. Five independent runs on a randomly sampled 50-task subset: Claude Code + Claude-4-Sonnet scores 16.00/16.00/20.00/20.00/22.00% across rounds, mean 18.80% (STD 2.40), Pass@5 26.00%; OpenHands + o4-mini scores 14.00/16.00/18.00/20.00/16.00%, mean 16.80% (STD 2.04), Pass@5 20.00%. The paper reads the low STD as evidence run-to-run variance does not affect agent ranking, while the Pass@5-over-mean gap shows retries surface additional solvable tasks. ↩
Cross-checked across four sources: DevOps-Gym paper Section 3.1 prose (54 build & config, 34 monitoring [29 synthetic + 5 real], 310 issue resolving, 310 test generation, 18 end-to-end, sums to 708 four-stage / 726 total); paper Section 1 introduction ("704 tasks... 14 end-to-end pipeline tasks," additive framing implies 718 total); paper Appendix Table 5 (48 build & config, 30 monitoring, 310 issue resolving, 310 test generation, no end-to-end row, sums to 698); and the ucsb-mlsec/DevOps-Gym README's own statistics table (48/30/310/310/14, summing to 712) against its own separately stated headline of "704 real-world DevOps tasks." No two of these four sources arithmetically agree; the 48/30/310/310 four-stage split (shared by Table 5 and the README) and the "14" end-to-end count (corroborated independently by Table 4's 14 listed rows and the README) are the best-supported figures. ↩↩↩↩↩
ucsb-mlsec/DevOps-Gym GitHub repository and README, verified via the GitHub REST API and raw README fetch on 2026-07-16: public, MIT license claimed in the README badge and Citation/License section, but no LICENSE file present at the repository root (404 on raw.githubusercontent.com/.../main/LICENSE); created 2026-02-01, last pushed 2026-03-11, 28 stargazers, 3 forks, 3 open issues; top-level contents README.md, collect/, decontamination/, requirements.txt, tasks/{build,end_to_end,issue_resolving,monitor,test_generation}/, utils/. README Quick Start: tb run --agent <agent-name> --model <model-name> --dataset-path "$(pwd)/tasks/<category>" --n-concurrent <n> via the Terminal-Bench tb CLI; per-task structure Dockerfile + task.yaml + setup.sh + evaluate.sh. ↩↩↩↩↩↩
Appendix D case studies. D.1 (Build): the MobArena Maven-to-Gradle migration task, where the agent's final ./gradlew test run reports "BUILD FAILED in 6s, 253 tests completed, 88 failed" after the agent otherwise completed the migration and produced correct artifacts; and the Caddy GoReleaser task, where the agent's generated .goreleaser.yml validates syntactically but omits the project-specific nobadger/nomysql/nopgx build tags needed for a functionally correct Caddy build. ↩↩