Skip to content
Markdown

SREGym: a live benchmark for AI SRE agents with high-fidelity failure scenarios

Scope: SREGym (arXiv:2605.07161, UIUC and University of Toronto, May 2026), a live SRE-agent benchmark built on real Kubernetes-native application stacks (DeathStarBench, Train Ticket, Astronomy Shop, plus in-house apps) with 90 curated SRE problems, a fault/noise injector library (50 fault primitives across 139 deployable services), and three failure modes: metastable, concurrent, and correlated failures. It covers the problem formalization, the checklist-based diagnosis oracle and state-based mitigation oracle, the evaluated results for Stratus, Claude Code, and Codex, and how to run or extend the benchmark honestly. The task-decomposition sibling this paper positions itself against is AIOpsLab; the field view of running an agentic ops loop in production is agentic AIOps; general agent-scoring discipline (datasets, rubrics, LLM-as-judge caveats) is evaluating agents; a production multi-agent incident-diagnosis pipeline you could point at this benchmark is OpsAgent.

Every number and case-study description on this page is the paper's own (arXiv:2605.07161v2, Sections 2-3 and Appendices A-H); none of SREGym's live-cluster runs were reproduced here, no cluster was stood up. The GitHub repository (SREGym/SREGym) was verified public, non-fork, MIT-licensed via the GitHub API on 2026-07-16, and its README installation/quickstart commands are quoted as reference templates, unexecuted. The Python block below is executed and asserted; it models the paper's Section 2.5 checklist-scoring formula (an equal-weighted, three-dimension aggregation with threshold tau=7/9) exactly as specified, not SREGym's actual LLM-as-judge pipeline, which needs a live judge model and real agent trajectories neither of which run here.

What it is

SREGym formalizes an SRE problem as a four-tuple P = (E, I, F, O): a system environment E (a real cloud-native application deployed on Kubernetes), an agent interface I (the tools an agent can call), a set of faults and noises F = {f1, ..., fk} (a subset of which are transient, low-impact noises rather than the target root cause), and a pair of oracles O = (Od, Om) for diagnosis and mitigation.1 Unlike benchmarks that decompose a failure into isolated detection/localization/RCA/mitigation sub-scores and grade each independently, SREGym evaluates the entire failure scenario holistically in one continuous session, arguing that real incident response is a single loop where earlier evidence and actions shape later ones.1

The system environment ships a catalog of real applications rather than toy services: DeathStarBench's SocialNetwork and HotelReservation, Train Ticket (40 microservices), the OpenTelemetry Astronomy Shop demo, and two in-house applications (a satellite-orbit simulator and a flight-booking service), each paired with backend data systems (MongoDB, TiDB, Kafka, MySQL, Valkey) and deployed via Helm.1 Agents interact through Model Context Protocol (MCP) servers for metrics (Prometheus), logs (Loki), traces (Jaeger), cluster control (kubectl), and submission, plus raw observability and Kubernetes API endpoints for agents that want to build their own tools; the only interface requirement placed on an agent is that it eventually calls submit(), which is what lets SREGym evaluate architecturally different agents (a multi-agent system, a ReAct loop, a general-purpose coding agent) against the same problem pool without porting them to a fixed action schema.1

Faults, noise, and failure modes. Table 1 of the paper lists the fault/noise mechanisms: killing a process or pod (fail-stop), stressing hardware (fail-slow), failing a syscall via eBPF (OS/hardware faults), corrupting a disk sector, mutating deploy.yaml or application/Kubernetes config, shipping buggy application code or a buggy operator, increasing client load, pausing/restarting unrelated pods, injecting network latency or packet drop, and stressing node resources (noisy neighbors).2 SREGym provides 50 such fault primitives compatible with 139 deployable services (plus 3 worker nodes for hardware-level faults), yielding 3,623 viable (fault, target) pairs before composing noises or multiple faults; the paper's own combinatorics (Table 7) break this down by compatibility class, e.g. 25 universal Kubernetes-level faults each apply to all 139 pods (3,475 pairs), while 4 MongoDB-specific faults apply only to the 18 MongoDB pods (72 pairs).2 The 90 curated, evaluated problems exercise only 90/3,623 = 2.48% of the viable pairs, a fact the paper reads as headroom for scaling the pool (and, longer term, as a natural reinforcement-learning rollout space) rather than as the benchmark's full extent.2

Noises are defined as transient, self-recovering disturbances (a pod crashing and rescheduling, a few dropped requests) injected on a configurable schedule by the framework itself, not hardcoded per problem; agents see symptoms from both the target fault and the noise and must tell them apart.3 Three failure modes structure the harder problems: metastable failures, where a transient trigger (e.g. an aggressive gRPC retry policy under a load surge) pushes the system into a self-sustaining degraded state that persists after the trigger is gone; concurrent failures, multiple independent faults injected simultaneously that the agent must prioritize by actual user-facing impact rather than treat as one problem; and correlated failures, multiple components failing together because they share a root cause the agent must trace back to.3

Evaluation oracles. The diagnosis oracle is a checklist-based LLM-as-a-judge: given a ground-truth root-cause description and the agent's submitted diagnosis, an LLM evaluator answers N=9 Yes/No questions grouped into K=3 equally-weighted dimensions (fault localization, fault characterization, failure scope), each dimension's score is the fraction of "yes" answers, the aggregate score is the equal-weighted sum, and the verdict is a pass if the aggregate clears a default threshold tau = 7/9.4 Validated against 100 stratified samples independently labeled by a domain expert, the default judge (Claude Sonnet-4.6) agrees with human labels at Cohen's kappa = 0.90, and two alternate LLM judges (GPT-5.4, Kimi K2.5) converge with each other at kappa = 0.94 while retaining substantial agreement with humans (kappa = 0.70 and 0.76 respectively), which the paper reads as evidence that the checklist decomposition, not the specific judge model, is doing the work.4 The mitigation oracle is problem-specific and state-based: it checks whether the target fault is actually resolved and the whole system, not just the injected resource, has returned to a healthy state, using both client-side (request success rate) and system-side (process, cluster) observability.4

Why use it

  • It measured a real capability gap, not a narrow one. Across three agents (Stratus, Claude Code, Codex) paired with three frontier models (Claude Sonnet-4.6, Kimi K2.5, GPT-5.4), diagnosis success ranges 38.9-72.6% and mitigation success 57.3-78.5%; Claude Code with Sonnet-4.6 has the highest end-to-end (both-correct) rate, Stratus with Sonnet-4.6 has the highest mitigation rate (the paper attributes this to Stratus's undo-and-retry mechanism), and Stratus with Kimi K2.5 trails every configuration, attributed to the weaker underlying model rather than the agent design.5
  • Token cost is decoupled from success on this benchmark. Stratus averages 0.53M tokens per run (0.41-0.81M across configurations); Claude Code uses about 3.0x that (1.59M) and Codex about 3.6x (1.93M), because the coding agents pull raw observability streams into the LLM context instead of preprocessing them, and the highest end-to-end score in the whole table is Stratus-adjacent rather than the highest-token configuration.8
  • Noise degrades diagnosis more than mitigation, and it does so through a specific, named failure pattern. Every agent-model pair's diagnosis rate drops under noise while mitigation is comparatively robust, because agents recover from a wrong hypothesis during mitigation more often than they avoid forming one during diagnosis; trajectory inspection shows a consistent greedy pattern, agents anchor on the first plausible anomaly (whether it is the injected noise or an unrelated pre-existing condition) and never revisit that anchor even when later evidence points elsewhere.10
  • The benchmark exposes exactly the gap it was built to expose. Problems ported from AIOpsLab/ITBench are comparatively easy (mitigation above 80% for Stratus-Sonnet); the new failure classes unique to SREGym (hardware faults, metastable interactions, OS-layer faults) are not: end-to-end success for Stratus-Sonnet, Claude Code, and Codex falls from 63.7% to 17.9%, from 60.8% to 28.2%, and from 57.8% to 15.4% respectively moving from the ported subset to the new-failures subset (clean condition).6 The paper's own headline framing, "up to 40% differences in end-to-end results" across failure categories, is looser than these per-agent gaps, which actually range about 32.6 to 45.8 percentage points; read the headline as a rounded summary, not a precise bound, and use Table 4's per-agent numbers when sizing a claim.6
  • Diagnosis-mitigation correlation is measured, not assumed. Conditioned on a correct diagnosis, mitigation succeeds 69-88% of the time across agent-model-noise configurations (P(M|D)); conditioned on an incorrect diagnosis, it still succeeds 35-59% of the time (P(M|¬D)), a gap of 23-34 percentage points per configuration. Some of that residual mitigation-without-diagnosis success is genuine self-correction (an agent observes the failure persisting and forms a new hypothesis mid-mitigation); some of it is pattern-matching a known symptom to a fix that happens to work without the agent ever understanding why.7

When to use it (and when not)

  • Use it to compare candidate SRE agents (or an agent upgrade) against real, live Kubernetes failures with ambient noise before granting write access to a cluster, the same pre-deployment discipline this KB recommends generically in evaluating agents.
  • Use it specifically when your evaluation needs to distinguish "solved a symptom" from "understood the failure": the diagnosis/mitigation split and the P(M|D) vs P(M|¬D) conditional numbers are built exactly for this, and they catch the pattern-matched-fix-without-understanding failure mode that a mitigation-only score hides.7
  • Use it to build custom problems for fault classes your own incident history contains: the fault library is parametric (a new problem is a Python class implementing inject_fault(), workload setup, and oracle wiring) and the 3,623-pair combinatorial space is designed to be extended, not just consumed.2
  • Disclose the authorship overlap before citing the agent comparison as fully independent. Four of SREGym's eight authors (Jackson Clark, Yiming Su, Yinfang Chen, Tianyin Xu) are also authors of Stratus (arXiv:2506.02009), one of the three agents SREGym evaluates, and Stratus with Sonnet-4.6 comes out with the highest mitigation success rate in Table 3. This is disclosed nowhere as a conflict in the paper itself; it does not mean the numbers are wrong (the diagnosis oracle's independent kappa validation is real, and Stratus does not win on end-to-end or diagnosis), but a claim built on "Stratus has the best mitigation rate" should carry this note.13
  • Do not treat a high score as evidence of production readiness. SREGym's largest application (Train Ticket) has 40 microservices against thousands in hyperscale production, its noise model covers routine pod churn but not high-variance traffic anomalies, partial network partitions, or gradual resource exhaustion, and the evaluated agent-model sample is 3x3 by budget constraint. The paper's own framing: read SREGym results as a lower bound on the failure modes an agent can recover from, not an upper bound on what it will encounter in production.11
  • Do not average away the low-level-stack collapse. On the paper's hardware-fault case study, no run across three agents scored a diagnosis above 0.22, and the fault-characterization dimension scored exactly 0 in every run; a blended average with easier application-layer problems would hide this completely.10

Architecture

The benchmark composes five pieces into one closed loop: agents talk to the system environment only through the MCP-based agent interface; fault and noise injectors, driven by a problem's spec, perturb the environment on a schedule the coordinator controls; a data collector exposes Prometheus/Loki/Jaeger telemetry back through that same interface; and an agent's two submissions (diagnosis, then mitigation-complete) each trigger their own oracle, both reporting to a leaderboard.1

flowchart TB
    AG["Agents under test:<br/>Stratus, Claude Code, Codex, ..."] -->|"MCP calls: get_metrics, get_logs,<br/>get_traces, kubectl, submit()"| AI["Agent Interface (MCP servers)"]
    AI --> ENV["System Environment (Kubernetes):<br/>DeathStarBench, Train Ticket, Astronomy Shop,<br/>in-house apps + MongoDB/TiDB/Kafka/MySQL/Valkey"]
    ENV --> DC["Data collector:<br/>Prometheus metrics, Loki logs, Jaeger traces"]
    DC --> AI
    PS["Problem Spec:<br/>system spec + fault spec + oracle spec"] --> COORD["Coordinator / Runtime<br/>(owns fault + noise scheduling)"]
    COORD -->|"scheduled, timed, hidden behind a proxy"| FN["Fault + noise injectors:<br/>eBPF syscall faults, disk-sector corruption,<br/>K8s/app misconfig, operator misoperation,<br/>load/latency noise"]
    FN --> ENV
    AG -->|"submit(diagnosis)"| OD["Diagnosis oracle:<br/>checklist LLM-as-judge, N=9 Q, K=3 dims, tau=7/9"]
    AG -->|"submit(mitigation done)"| OM["Mitigation oracle:<br/>state-based whole-system health check"]
    OD --> LB["Leaderboard"]
    OM --> LB

The diagnosis oracle's checklist score (executed)

The scoring formula in Section 2.5 is exact enough to implement directly: per-dimension score s_k is the fraction of "yes" answers among that dimension's 3 questions, the aggregate S = sum_k (1/3) * s_k, and the verdict is S >= tau with tau = 7/9. The paper states this threshold "forbids a submission to pass while missing an entire dimension"; the code below checks that claim as a structural property rather than taking it on faith, and also verifies a consequence the paper does not spell out: because all three dimensions carry equal weight over equal question counts, S collapses to plain total-question accuracy, so two submissions with the same total yes-count but a different per-dimension spread land on the identical score. That equivalence is this page's own derivation from the stated formula, not a claim the paper makes.

# sregym_oracle_model.py - executed: SREGym's checklist-based diagnosis oracle (Sec. 2.5).
# N=9 Yes/No questions over K=3 equally-weighted dimensions (fault localization, fault
# characterization, failure scope), aggregated as S = sum_k w_k * s_k with s_k the fraction
# of "yes" answers in dimension k, verdict = 1[S >= tau], default tau = 7/9. This is a model
# of the scoring formula itself (paper's exact equations and threshold), not a rerun of the
# paper's LLM-as-judge pipeline (which needs an LLM evaluator and real trajectories).
from __future__ import annotations

from fractions import Fraction

DIMENSIONS = ("localization", "characterization", "scope")
TAU = Fraction(7, 9)


def score(answers: dict[str, list[int]]) -> tuple[dict[str, Fraction], Fraction, bool]:
    """answers: dimension name -> list of 0/1 judgments (3 questions per dimension in the
    paper). Returns (per-dimension score, aggregated S, verdict)."""
    assert set(answers) == set(DIMENSIONS), "must answer every dimension, none skipped"
    assert all(len(v) == 3 for v in answers.values()), "paper fixes 3 questions/dimension"
    assert all(y in (0, 1) for v in answers.values() for y in v), "each judgment is Yes/No"
    s = {d: Fraction(sum(v), len(v)) for d, v in answers.items()}
    w = Fraction(1, len(DIMENSIONS))  # equal weights, paper Sec 2.5: w_k = 1/3
    S = sum(w * s[d] for d in DIMENSIONS)
    return s, S, S >= TAU


# 1. Perfect diagnosis: every question answered Yes in every dimension.
s, S, ok = score({"localization": [1, 1, 1], "characterization": [1, 1, 1], "scope": [1, 1, 1]})
assert S == 1 and ok

# 2. Adversarial: the agent aces two dimensions completely but says nothing useful in the
# third (0/3). Naive question-level accuracy is 6/9 = 66.7%, which superficially looks close
# to a passing grade, but the paper's own claim is that tau=7/9 "forbids a submission to pass
# while missing an entire dimension" -- this is the exact boundary that claim rests on.
s, S, ok = score({"localization": [1, 1, 1], "characterization": [1, 1, 1], "scope": [0, 0, 0]})
assert S == Fraction(2, 3) and not ok, (s, S, ok)
assert S < TAU, "zeroing one whole dimension must fail even with two perfect dimensions"

# 3. Exact threshold boundary: 7 of 9 questions correct, distributed as two perfect
# dimensions plus one partial hit (1/3) in the third -- S lands exactly on tau and passes
# (paper's verdict rule is >=, not >).
s, S, ok = score({"localization": [1, 1, 1], "characterization": [1, 1, 1], "scope": [1, 0, 0]})
assert S == TAU and ok, (s, S, ok)

# 4. Same 7-of-9 raw yes-count, distributed as (3,2,2) across dimensions instead of case 3's
# (3,3,1): with equal question counts per dimension the equal weighting makes S depend only
# on the total yes-count, not on which dimension the yeses land in, so both distributions
# land on the exact same S and the same pass verdict.
s2, S2, ok2 = score({"localization": [1, 1, 1], "characterization": [1, 1, 0], "scope": [1, 1, 0]})
assert S2 == TAU and ok2
assert S2 == S

# 5. Edge case: an empty-content diagnosis (every question No) scores S=0 and fails cleanly,
# no division-by-zero or vacuous pass.
s, S, ok = score({"localization": [0, 0, 0], "characterization": [0, 0, 0], "scope": [0, 0, 0]})
assert S == 0 and not ok

# 6. Malformed input is rejected rather than silently scored: a dimension with the wrong
# question count, and a submission missing a dimension entirely.
try:
    score({"localization": [1, 1], "characterization": [1, 1, 1], "scope": [1, 1, 1]})
    raise AssertionError("must reject a dimension with != 3 questions")
except AssertionError as e:
    assert "3 questions" in str(e)

try:
    score({"localization": [1, 1, 1], "characterization": [1, 1, 1]})
    raise AssertionError("must reject a submission missing a dimension")
except AssertionError as e:
    assert "every dimension" in str(e)

print("oracle model OK: tau=7/9 =", float(TAU),
      "| two-perfect-one-zero S =", float(Fraction(2, 3)), "(fails)",
      "| threshold-boundary S =", float(TAU), "(passes)")

Executed output:

oracle model OK: tau=7/9 = 0.7777777777777778 | two-perfect-one-zero S = 0.6666666666666666 (fails) | threshold-boundary S = 0.7777777777777778 (passes)

How to run it against your own agent

The repository (SREGym/SREGym, MIT-licensed, verified public on 2026-07-16) provides a push-button setup for exactly this. These commands are quoted from the repo's README as of that date; treat them as reference templates and verify against the current repo before running, they were not executed as part of writing this page.12

# Reference template (README, retrieved 2026-07-16). Not executed here.
git clone --recurse-submodules https://github.com/SREGym/SREGym
cd SREGym
uv sync
uv run prek install

# Cluster: either a self-managed K8s cluster via the bundled Ansible playbook
# (scripts/ansible/README.md, needs SSH/root on the target hosts -- a managed
# K8s service will not work, SREGym needs OS-level node configuration), or a
# local emulated cluster via kind:
bash kind/setup_kind_cluster.sh x86   # or: arm

# Run the bundled Stratus agent against the problem pool:
export ANTHROPIC_API_KEY="sk-ant-..."
python main.py --agent stratus --model anthropic/claude-sonnet-4-6

# Or OpenCode against a local model via Ollama (uses the endpoint's
# OpenAI-compatible /v1 API; judge defaults to the agent model unless
# --judge-model is set separately):
ollama pull qwen3-coder:30b
export AGENT_API_BASE="http://127.0.0.1:11434/v1"
python main.py --agent opencode --model local/qwen3-coder:30b

Two operational details worth knowing before wiring your own agent in: agents always run inside an isolated Docker container that has no access to problem definitions or grading logic, which is the mechanism behind the paper's reward-hacking protection (Appendix B); and model selection goes through LiteLLM model strings directly, so any LiteLLM-supported provider (OpenAI, Anthropic, Gemini, Bedrock, Azure, or an OpenAI-compatible local server) works without a config file.12

How to extend it with new problems and fault scenarios

A new problem is a Python class implementing four things: an application to deploy, a fault injector call, a diagnosis oracle (typically LLMAsAJudgeOracle against a written root-cause description), and a mitigation oracle. The paper's own worked example (Figure 3, quoted as a reference template, unexecuted):2

# Reference template (paper Figure 3). Not executed here; verify current API on the repo.
class K8sNetworkPortMisconfig(Problem):
    def __init__(self):
        self.app = SocialNetwork()
        self.app.create_workload()
        self.root_cause = ("The user-service has a misconfigured network port [...]")
        self.diagnosis_oracle = LLMAsAJudgeOracle(problem=self, expected=self.root_cause)
        self.mitigation_oracle = MitigationOracle(problem=self)

    @mark_fault_injected
    def inject_fault(self):
        injector = NetworkPortFaultInjector(namespace=self.namespace)
        injector._inject(fault_type="port-misconfig", microservice="user-service")

Noises are injected by the framework itself, not authored per problem, so a new problem's job is limited to the target fault and its oracle wiring.2 Four engineering constraints from Appendix B are worth internalizing before writing new problems, because each one is a documented departure from prior benchmarks the authors evaluated against:9

  • Simulate faults, not symptoms. Chaos-engineering tools (Chaos Mesh, Chaosblade) inject observable symptoms (killed pods, dropped packets, throttled CPUs) with no underlying defect; the only "mitigation" for a symptom-only fault is stopping the chaos tool, which is not an operational skill worth rewarding. The paper names three prior benchmarks that conflate the two: MicroRemed sources its faults entirely from Chaos Mesh, Cloud-OpsBench draws on Chaosblade alongside K8s misconfigurations, and even ITBench (which otherwise injects faults directly) wraps a Chaos Mesh schedule for 6 of 36 scenarios (about 16%).9
  • Hide the fault-injection plane. If fault-injector processes are visible as identifiable pods in the environment the agent inspects, discovering and disabling them becomes a viable exploit; SREGym hides its injection plane behind a proxy the agent cannot see.9
  • Grade on state, not on alert-clearing. The Stratus paper found that 8 of 18 ITBench mitigation problems (44%) can be "solved" by a generic pod-restart loop where the fault injector loses track of the restarted pod, the alert clears, and the agent is credited without ever touching the actual defect. SREGym's mitigation oracle instead probes live system state.9
  • Build a programmable runtime, not a script pipeline. Metastable-failure problems need an infrastructure constraint and an application-level trigger injected with specific timing and ordering, and noise injection needs its own concurrent schedule running alongside the target fault; Ansible-script-based runtimes (the paper names ITBench) cannot express this coordination directly and tend to defer it to ad hoc shell scripts. SREGym runs fault scheduling, noise scheduling, and oracle probing as a single Python process for exactly this reason.9

If you're extending the fault library itself rather than composing existing primitives, the combinatorial coverage table (Appendix C, Table 7) is the map: which fault classes are universal (apply to every pod) versus scoped to a storage-dependent, operator-level, or database-specific subset of the 139-service target space, and how many viable pairs each class contributes toward the 3,623 total.2

How to interpret and report scores honestly

  • Report diagnosis, mitigation, and end-to-end separately, and read the gap. A mitigation-only number hides that 35-59% of mitigations succeed on an incorrect diagnosis (P(M|¬D)); some of that is genuine self-correction during a multi-attempt mitigation, some is symptom pattern-matching with no causal understanding. If your use case cares whether the agent understood the failure (not just whether the cluster ended up healthy), P(M|D) vs P(M|¬D) is the number to track, not the mitigation rate alone.7
  • Partition by problem provenance before quoting a headline number. The paper's own Table 4 split (problems ported from AIOpsLab/ITBench, similar new problems sharing a fault family, and failures unique to SREGym) shows every agent's end-to-end rate falling 33 to 46 percentage points from the easiest to the hardest partition. A single averaged score across all 90 problems blends an easy, mostly-solved subset with a hard subset no agent clears at even 30%; report the partition, or at minimum flag which subset a quoted number came from.6
  • Report the clean and noisy conditions separately. Diagnosis degrades under noise for every agent-model pair while mitigation is comparatively robust; treating a noisy-condition diagnosis drop as an agent regression, rather than the expected effect of the noise model, will misattribute the cause. Table 3's □/■ split exists precisely so these don't get collapsed into one number.5
  • Don't read TTD/TTM as latency without checking the cap. Time-to-Diagnose and Time-to-Mitigate are capped at the 1800-second agent timeout, and timed-out runs contribute the cap value into the mean rather than being dropped, which is a deliberate choice to avoid survivorship bias but means a high mean TTM can reflect either genuinely slow mitigation or a cluster of timeouts; check the underlying success rate alongside the time metric.5
  • Don't average away a total capability gap. On the hardware-fault case study, no run across three agents and 9 total trials scored a diagnosis above 0.22, and the fault-characterization checklist dimension scored exactly 0 in every single run; a benchmark-wide average absorbs this into a small dent rather than surfacing a class of failure the evaluated agents essentially cannot diagnose at all. Report per-fault-class breakdowns, not just the aggregate, especially for lower-stack faults (hardware, OS, kernel) that the paper explicitly calls out as underrepresented in prior benchmarks.10
  • Version the problem pool and disclose author overlap where relevant. The evaluated 90 problems are 2.5% of the 3,623 viable fault-target pairs and the pool is stated to grow; cite a pool revision or count alongside any score the way AIOpsLab's own maintenance guidance recommends. And if you cite the specific claim that Stratus has the best mitigation rate among the three evaluated agents, note that the SREGym and Stratus author lists overlap by four names.213

Failure modes

Pitfall Cause Fix
Diagnosis fixates on the first plausible anomaly Agents treat the first-observed anomaly, whether injected noise or an unrelated pre-existing condition (e.g. a low memory limit), as the target failure and stop generating competing hypotheses, even after later evidence points elsewhere.10 Score per-dimension (localization/characterization/scope), not just pass/fail; treat greedy anchoring as a named error-taxonomy category in trajectory review, not noise.
Hardware/OS-layer faults score near zero Agents reason only about application-layer error handling and never suggest disk-level diagnostics (dmesg, smartctl) even when the symptom is an I/O error propagating from a physical disk fault.10 Include lower-stack fault classes in your own eval mix explicitly; do not let a benchmark-wide average hide a complete capability gap on one fault layer.
Metastable failures diagnosed only partially Agents reliably find the visible application-level trigger but almost never discover the invisible infrastructure constraint (a resource quota) that sustains the degraded state, and in the one run that did find it, the agent dismissed the trigger as an unrelated artifact instead of connecting the two.10 Require the diagnosis to name every interacting component; the checklist's scope-precision dimension (D3) is designed to catch a partial, one-sided answer.
Mitigation succeeds without correct diagnosis Pattern-matching a known symptom to a fix, or iterative retry-until-the-failure-clears, both restore system health without the agent understanding the root cause (P(M|¬D) = 35-59%).7 Report P(M|D) and P(M|¬D) separately; do not credit mitigation-only success as evidence of diagnostic capability.
Reward hacking via injector discovery or alert-clearing Fault-injection processes visible as identifiable pods let an agent disable them instead of fixing the fault; oracles that key on alert suppression credit a generic pod-restart loop that never touches the actual defect (documented at 44% of ITBench mitigation problems in the Stratus paper).9 Hide the fault-injection plane behind a proxy invisible to the agent; grade mitigation against live system state, not alert status.
Symptom-only "faults" with no real defect Reusing chaos-engineering tools (Chaos Mesh, Chaosblade) that inject symptoms (killed pods, throttled CPU) rather than root causes; the only valid "fix" is stopping the chaos tool, which is not an operational skill.9 Inject faults via direct Kubernetes manipulation, eBPF syscall faults, or config mutation, so a real defect exists for the agent to find and fix.
Noise-induced diagnosis drop misread as an agent regression Ambient noise genuinely reduces diagnosis success for every agent-model pair while mitigation stays comparatively robust; this is the noise model working as intended, not an agent getting worse.5 Always report clean and noisy conditions separately (the paper's own □/■ split); don't compare a noisy run against a clean baseline without flagging the condition.
LLM-judge oracle variance across evaluator models The diagnosis oracle is itself an LLM evaluator; different judge models agree with human labels at different kappa (0.70-0.90) and with each other more than with humans on some pairs.4 Use one fixed judge model consistently within any comparison you intend to report; don't mix judge models across runs being compared head to head.
Testbed-to-production scale gap SREGym's largest application (Train Ticket) has 40 microservices against thousands of interacting services in hyperscale production; scale affects both how many candidate services an agent must search and how many cross-service dependencies a mitigation must reason about.11 Treat a high SREGym score as a lower bound on recoverable failure modes, not proof of production readiness, exactly as the paper's own broader-impacts section frames it.
Growing the pool without tracking a revision 90 curated problems are 2.5% of 3,623 viable fault-target pairs, and the pool is explicitly stated to keep growing; adding problems shifts every aggregate.2 Version and report the problem-pool count or commit alongside any score, the same discipline AIOpsLab recommends for its own growing pool.

References

  • Clark, Su, Pial, Tian, Gniedziejko, Jacobsen, Chen, Xu, "SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios," arXiv:2605.07161: https://arxiv.org/abs/2605.07161
  • SREGym repository (verified public, non-fork, MIT-licensed, 2026-07-16): https://github.com/SREGym/SREGym
  • Chen, Pan, Clark, Su, Zheutlin, Bhavya, Arora, Deng, Jha, Xu, "STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds," arXiv:2506.02009: https://arxiv.org/abs/2506.02009
  • Chen, Shetty, Somashekar, Ma, Simmhan, Mace, Bansal, Wang, Rajmohan, "AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds" (MLSys'25), arXiv:2501.06706: https://arxiv.org/abs/2501.06706
  • Jha et al., "ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks" (ICML'25), arXiv:2502.05352: https://arxiv.org/abs/2502.05352
  • Gan et al., "DeathStarBench: An Open-Source Benchmark Suite for Microservices and Their Hardware-Software Implications for Cloud & Edge Systems" (ASPLOS'19): https://github.com/delimitrou/DeathStarBench
  • OpenTelemetry Astronomy Shop demo (otel-demo): https://github.com/open-telemetry/opentelemetry-demo

Related: AIOpsLab · Agentic AIOps · Evaluating agents · Agentic incident management (OpsAgent) · Evaluation integrity and anti-gaming · Multi-agent collaboration · SRE, platform and MLOps practices

  1. Clark et al., arXiv:2605.07161v2. Problem tuple P=(E,I,F,O) and the holistic-versus-decomposed evaluation argument: Section 2.1 and footnote 3. System environments (DeathStarBench, Train Ticket 40 microservices, Astronomy Shop, in-house satellite-orbit simulator and flight-booking apps; MongoDB/TiDB/Kafka/MySQL/Valkey backends; Helm deployment): Section 2.2. Agent interface (MCP servers for metrics/logs/traces/cluster-control/submission; submit() as the only required call; no architectural assumption on the evaluated agent): Section 2.3. Overall framework components: Figure 2 and Section 2 preamble. ↩↩↩↩↩

  2. Clark et al., arXiv:2605.07161v2, Section 2.4, Table 1 (fault/noise mechanisms and the failures they simulate) and Figure 3 (the K8sNetworkPortMisconfig problem-definition example, quoted verbatim). Combinatorial coverage: Appendix C, Table 7 (50 fault primitives, 139 deployable services plus 3 worker nodes, 3,623 viable fault-target pairs by compatibility class: universal K8s-level 25 faults x 139 pods = 3,475 pairs, storage-dependent 5x6=30, DaemonSet-level 1x3=3, operator-level 6x5=30, MongoDB-specific 4x18=72, Valkey-specific 2x1=2, app-layer misconfiguration 1x2=2, node/kernel 3x3=9); "the 90 curated problems we evaluate against in Section 3 exercise only 2.5% of the 3,623 viable (fault, target) pairs" (90/3623 = 2.48%, computed here from the paper's own totals). ↩↩↩↩↩↩↩↩↩

  3. Clark et al., arXiv:2605.07161v2, Section 2.4 (noise definition as transient, self-recovering, framework-scheduled disturbances) and the three failure-mode paragraphs (metastable, concurrent, correlated failures). Figure 4a's metastable example (Hotel Reservation, DeathStarBench: aggressive gRPC connection timeout 50ms and retry count 30, 3000 req/s load, triggered by a transient CPU stress) is not named in Section 2.4 itself; the repository's Problem List.md describes capacity_decrease_rpc_retry_storm ("RPC module is misconfigured, then a CPU containment will trigger a retry storm") as the closest match, which is a distinct problem from gc_capacity_degradation (a GOGC/garbage-collection-based metastable failure, see 10, Appendix D.1) -- the two names are not interchangeable. Figure 4b is the Social Network concurrent-failure example (scheduler misconfiguration on an observability pod versus a network-port misconfiguration on UserID). Figure 4 illustrates both. ↩↩

  4. Clark et al., arXiv:2605.07161v2, Section 2.5. Checklist formula (N=9 Yes/No questions, K=3 equally-weighted dimensions w_k=1/3, per-dimension score s_k as the fraction of yes answers, aggregate S = sum_k w_k s_k, verdict v_hat = 1[S >= tau], default tau = 7/9, "forbids a submission to pass while missing an entire dimension"); the three dimensions (fault localization, fault characterization, failure scope) and their questions in full: Appendix A, Table 6. Inter-evaluator validation on 100 stratified samples: Table 2 (Sonnet-4.6 vs Human agreement 0.95, kappa 0.90; Sonnet-4.6 vs GPT-5.4 kappa 0.76; Sonnet-4.6 vs K2.5 kappa 0.82; GPT-5.4 vs K2.5 kappa 0.94; GPT-5.4 vs Human kappa 0.70; K2.5 vs Human kappa 0.76). Mitigation oracle (problem-specific, state-based, checks whole-system health via client- and system-side observability): Section 2.5, "Mitigation Oracle" paragraph. ↩↩↩↩

  5. Clark et al., arXiv:2605.07161v2, Section 3 and Table 3 (overall results by agent/model/noise condition: Stratus+Sonnet-4.6 diag 61.5%/51.5%, mitig 78.5%/65.5%, E2E 54.8%/40.2% clean/noisy; Stratus+K2.5 diag 41.3%/38.9%, mitig 60.6%/57.3%, E2E 32.9%/30.4%; Claude Code+Sonnet-4.6 diag 72.6%/62.6%, mitig 75.6%/76.3%, E2E 60.7%/53.7%; Codex+GPT-5.4 diag 70.0%/59.3%, mitig 65.2%/64.0%, E2E 53.3%/45.9%; table caption: TTD/TTM capped at the 1800s agent timeout, timed-out runs contribute the cap value into the mean). "Impact of Noises" paragraph: diagnosis success drops for every agent-model pair under noise, mitigation more robust than diagnosis, agents take a "greedy approach" anchoring on the first plausible anomaly (detailed in Appendix D). ↩↩↩↩

  6. Clark et al., arXiv:2605.07161v2, Section 3.2 and Table 4 (results partitioned into Ported n=34, Similar Failures n=43, New Failures n=13; end-to-end clean-condition figures: Stratus+Sonnet-4.6 63.7% to 17.9%, Claude Code 60.8% to 28.2%, Codex 57.8% to 15.4%, Ported to New Failures). Abstract and Section 1: "up to 40% differences in end-to-end results" across failure categories (the per-agent Table 4 gaps computed here, 63.7-17.9=45.8, 60.8-28.2=32.6, 57.8-15.4=42.4 points, are the underlying basis for that rounded headline). ↩↩↩

  7. Clark et al., arXiv:2605.07161v2, Section 3.3 and Table 5 (conditional mitigation probability by diagnosis outcome, P(M|D) range 0.690-0.880 i.e. 69-88%, P(M|¬D) range 0.351-0.588 i.e. 35-59%, per-configuration gap 23-34 points computed here from Table 5's rows) and Table 12 (Stratus-only mitigation-attempt counts conditioned on diagnosis correctness, e.g. Stratus+Sonnet-4.6 clean: 1.88 attempts when diagnosis correct vs 3.82 when incorrect). ↩↩↩↩

  8. Clark et al., arXiv:2605.07161v2, Section 3.4 (tool-call category breakdown, Figure 5) and Appendix E (Tables 8-11, subcommand breakdowns) plus Appendix F (Figure 6, mean total tokens per run: Stratus 0.53M average / 0.41-0.81M range, Claude Code 1.59M / 3.0x Stratus, Codex 1.93M / 3.6x Stratus; token spend does not predict end-to-end success in this set). ↩

  9. Clark et al., arXiv:2605.07161v2, Appendix B ("Engineering Practices"). Live-environment rationale versus static snapshots. Agent-architecture-agnostic interface versus AIOpsLab's fixed ReAct-mediated orchestrator. Programmable single-process runtime versus Ansible-script substrates (named: ITBench) that cannot express metastable-failure timing or concurrent noise scheduling. Avoiding chaos-tool misuse (named: MicroRemed sources faults entirely from Chaos Mesh; Cloud-OpsBench draws on Chaosblade; ITBench wraps a Chaos Mesh schedule for 6 of 36 scenarios, about 16%). Reward-hacking protection (fault-injection plane hidden behind a proxy; state-based mitigation oracle; the Stratus-paper finding that 8 of 18, 44%, of ITBench mitigation problems are solvable by a generic pod-restart loop that clears the alert without addressing the defect). ↩↩↩↩↩↩↩

  10. Clark et al., arXiv:2605.07161v2, Appendix D. D.1 (metastable failure case study, gc_capacity_degradation: an aggressively low GOGC setting in a capacity-constrained namespace; example run where the agent fixated on a ~300ms trace gap between frontend and profile-service and diagnosed a fabricated tc netem network-latency rule instead of inspecting GOGC, never restarting the workloads that would have cleared the metastable loop). D.2 (hardware fault case study, latent_sector_error: intermittent pread() errors on a node's storage disk, MongoDB crashing with "read: input/output error"; across 3 Stratus, 3 Claude Code, and 3 Codex runs with no noise, no run scored a diagnosis above 0.22 and the fault-characterization dimension, D2, scored 0 in every run; agents blamed memcached connection drops or application error handling, never disk-level diagnostics such as dmesg or smartctl). D.3 (greedy-approach case study: the missing_env_variable_astronomy_shop problem from Figure 1, where Stratus latched onto an unrelated low memory limit and Claude Code onto Jaeger tracing infrastructure, neither ever inspecting the frontend component with the actually-missing environment variable; and the valkey_auth_disruption problem, where agents observed a TCP-health-check-succeeds-but-app-layer-connection-fails pattern consistent with an authentication failure, sometimes named the requirepass command as a candidate, but defaulted to a low-memory explanation anyway). ↩↩↩↩↩↩↩

  11. Clark et al., arXiv:2605.07161v2, Appendix H ("Limitations"): oracle variance (LLM-evaluator source of noise, mitigated by the structured rubric and per-dimension reporting), noise modeling (covers routine pod churn/resource stress, not high-variance traffic anomalies, partial network partitions, or gradual resource exhaustion), system scale (largest app Train Ticket at 40 microservices versus thousands in hyperscale production), environment scope (Kubernetes cloud-native only, not monolithic or edge deployments), agent coverage (3 agents x 3 models, budget-constrained, "correspondingly tentative" conclusions). Appendix G ("Broader Impacts"): the explicit framing that SREGym results should be read "as a lower bound on the failure modes an agent can recover from, not an upper bound on the failure modes it will encounter in real-world production deployments." ↩↩

  12. SREGym GitHub repository, https://github.com/SREGym/SREGym, README retrieved 2026-07-16 (repository confirmed public, non-fork, MIT-licensed via the GitHub REST API on the same date). Installation (git clone --recurse-submodules, uv sync, uv run prek install), cluster setup (self-managed via a bundled Ansible playbook requiring SSH/root on target hosts, or emulated locally via kind), quickstart (python main.py --agent stratus --model gpt-5), container isolation (agents run in an isolated Docker container rebuilt on --force-build, with no access to problem definitions or grading logic), and LiteLLM-based model selection (--model, --judge-model, provider examples for OpenAI/Anthropic/Google/Bedrock/Azure and local Ollama/vLLM/LM Studio endpoints) are all quoted from this README as of the retrieval date. ↩↩

  13. Chen, Pan, Clark, Su, Zheutlin, Bhavya, Arora, Deng, Jha, Xu, "STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds," arXiv:2506.02009 (NeurIPS'25), https://arxiv.org/abs/2506.02009: Transactional No-Regression (TNR) safety specification and the undo-and-retry mechanism SREGym's Section 3.1 attributes Stratus's mitigation-rate advantage to; author list cross-checked against SREGym's author list via the arXiv API on 2026-07-16 (Clark, Su, Chen, Xu appear on both papers, 4 of SREGym's 8 authors). Neither paper's author list designates a corresponding author anywhere (checked against the arXiv API metadata and the rendered PDF title page for both papers); SREGym's only author marker is "*Equal contribution" for Clark and Su. ↩↩