Markdown

Cloud-OpsBench: a reproducible State Snapshot benchmark for agentic RCA¶

Scope: Cloud-OpsBench (arXiv 2603.00468, CUHK/Sun Yat-sen University), a 452-case, 40-fault-type benchmark for agentic Root Cause Analysis (RCA) on Kubernetes that replaces live-cluster evaluation with a deterministic "State Snapshot" digital twin, its three-phase generation pipeline (fault-knowledge-base construction, closed-loop fault injection with Generator/Executor/Verifier agents, and snapshot freezing), its process-centric metrics (trajectory alignment, tool relevance/coverage, invalid-action and zero-tool-diagnosis rates), and the seven-model evaluation results. The live-cluster counterpart this benchmark is positioned against is AIOpsLab; the field view of running agentic operations on a cluster (task taxonomy, guardrails, evaluation discipline) is agentic AIOps; a worked production multi-agent RCA system this benchmark's findings would gate is OpsAgent.

Independently verified/executed here: the repository at https://github.com/LLM4Ops/Cloud-OpsBench is confirmed public as of 2026-07-16 (GitHub API returns "private": false), and the Python trajectory-alignment model below (Exact/In-Order/Any-Order Match, Tool Relevance, Tool Coverage) is executed and asserted against synthetic cases, including two adversarial cases (a causal-order-violating "lazy leap" trajectory and a schema-mismatched action), not against the live benchmark. All numbers in "Why use it" and the results discussion are quoted from the paper's tables and sections, each with a footnote pointing at the specific table/section; none were reproduced by running the benchmark itself, since only the dataset and harness code are public, not a hosted leaderboard I could query.

What it is¶

Cloud-OpsBench is a benchmark that formalizes agentic RCA as a trajectory-based decision process, f: <A_alert, E_snapshot> -> <T, R_hat>: an agent receives a natural-language alert and a cached system snapshot, then produces both a diagnostic trajectory (the sequence of thought/action/observation steps it took) and a predicted root-cause tuple, and both are scored.¹ Its central design contribution is the State Snapshot Paradigm: instead of deploying a live Kubernetes cluster and letting an agent query a volatile API server, the benchmark pre-executes an exhaustive parameter sweep of its ten diagnostic tools against a frozen fault state, so every plausible kubectl-style query the agent could issue resolves to a pre-recorded, deterministic answer, on average 487 distinct tool invocations get pre-computed per fault case.² Interactivity is reconstructed through a mocked operational interface: valid queries hit a deterministic entry, invalid queries (nonexistent pod names, wrong namespaces) map to realistic "Not Found" errors, so the agent still has to generate syntactically correct, semantically targeted commands even though nothing it does can perturb a real cluster.²

The dataset itself comprises 452 fault cases across 40 fault types organized into 7 categories (Admission Control, Scheduling, Startup, Runtime, Service Routing, Performance, Infrastructure) spanning the full Kubernetes stack, built on a Google Online Boutique deployment (11 microservices) on a 4-node Kubernetes v1.31 cluster with Prometheus and Istio for telemetry.³ Each case is generated by a three-agent pipeline (Generator, Executor, Verifier) that closes the loop between abstract fault knowledge and a physically-verified injection, then annotates ground truth as <R*, T*>: a structured root-cause tuple and a canonical diagnostic path synthesized by inverting the injection logic.⁴

Why use it¶

Zero-cost, fully reproducible replay. By decoupling state storage from runtime execution, a full benchmark run drops from the "often hours per run" the paper attributes to live-cluster evaluation to seconds on a standard laptop, removing the industrial-cluster barrier to entry the paper explicitly targets.⁶
It scores the investigation, not just the guess. Trajectory alignment (Exact/In-Order/Any-Order Match) and tool-usage metrics (Relevance, Coverage) catch an agent that is "right for the wrong reasons," which pure outcome accuracy (A@k) cannot distinguish from systematic reasoning.⁸
The results overturn the "shorter is better" intuition. DeepSeek-V3.2 posts the best outcome accuracy (A@1 0.73, A@3 0.79) with the longest trajectories (10.0 steps) and highest Coverage (0.88), while GPT-4o converges fastest (MTTI 23.27s, 5.67 steps) but scores lowest A@1 (0.49) among the six evaluated frontier/near-frontier models; the paper reads this as a "redundancy paradox" where DeepSeek-V3.2's 11% Redundant Action Rate functions as self-correction, not waste.⁹
It isolates a real bottleneck in small models. Qwen3-14B matches GPT-4o's Tool Relevance (0.63 vs 0.63) and nearly matches GPT-5's (0.65), meaning it knows which tool to call, but its Invalid Action Count (0.40) runs about 10x GPT-5's (0.04), and its A@1 (0.34) trails GPT-5 (0.67) by 33 points; the gap is malformed API calls and hallucinated parameters, not diagnostic judgment.⁹
In-context demonstrations, not manuals, close the gap. Retrieving 3 historical diagnostic traces (ICL) lifted Qwen3-14B's A@1 from 0.34 to 0.71 (matching unprompted GPT-5-tier performance) by cutting its Invalid Action Count from 0.40 to 0.29, while RAG over Kubernetes documentation only reached 0.50; procedural examples fix syntax, declarative docs do not.¹⁰

When to use it (and when not)¶

Use it to A/B test candidate RCA agents (model swaps, prompting strategies, tool-schema changes) under bit-for-bit identical fault conditions, something a live cluster cannot guarantee because of network jitter and scheduling nondeterminism.⁷
Use it as a pre-deployment gate the way AIOpsLab is used: check Tool Relevance/Coverage and Invalid Action Count before trusting an agent with real tool access, since the paper shows outcome accuracy alone hides syntactic fragility.
Use it as a training-data engine: the same pre-computed trajectories that power evaluation are the "harvested" demonstrations the paper uses for ICL and proposes for SFT bootstrapping of small models.⁶
Do not treat it as a remediation or intervention benchmark. The paper explicitly scopes RCA to reasoning over frozen, post-mortem evidence; it does not exercise mitigation actions or trial-and-error environmental intervention, unlike AIOpsLab's mitigation task level.¹³
Do not assume the fault taxonomy transfers past Kubernetes. The tools (kubectl-shaped) and fault categories are Kubernetes-specific; the authors themselves flag that monolithic or serverless architectures are out of scope, though they argue the underlying reasoning patterns (resource contention, cascading failure) generalize.¹³
Do not cite the seven-model leaderboard as current capability. It is a point-in-time snapshot (GPT-5/GPT-4o/Claude-4-Sonnet/DeepSeek-V3.2/Qwen3-235B/Qwen3-14B/Qwen3-8B as of the paper's writing); re-run it yourself before making a procurement or architecture decision.

Architecture¶

flowchart TB
  subgraph P1["Phase 1: Fault Knowledge Base Construction"]
    SRC["K8s docs, Stack Overflow,<br/>academic papers"] --> GEM["Gemini 3 Pro: extract<br/>structured metadata"]
    GEM --> DRAFT["Draft fault metadata:<br/>semantic + <P,A,S> tuple"]
    DRAFT --> JUDGE["LLM-as-Judge refine"]
    JUDGE --> EXPERT["Human expert review"]
    EXPERT --> KB["Validated Fault<br/>Knowledge Base<br/>(40 fault types)"]
  end

  subgraph P2["Phase 2: Automatic Fault Case Generation (closed loop)"]
    KB --> GENA["Generator Agent:<br/>CoT -> fault plan M=<P,A,S>"]
    CLUSTER["Target Kubernetes cluster<br/>(Online Boutique, 4 nodes)"] --> GENA
    GENA --> EXECA["Executor Agent:<br/>kubectl / ChaosBlade injection"]
    EXECA --> VERA["Verifier Agent:<br/>telemetry check"]
    VERA -->|"fault masked/not triggered"| GENA
    VERA -->|"verified"| SNAP["State Snapshot:<br/>exhaustive T1-T10 sweep,<br/>~487 tool calls/case"]
    SNAP --> GT["Ground truth <R*,T*>:<br/>injection logic inverted"]
  end

  subgraph P3["Phase 3: Benchmarking"]
    GT --> CASES["452 fault cases"]
    SNAP --> MOCK["Mocked tool interface<br/>(deterministic replay)"]
    CASES --> AGENT["Agent under test"]
    MOCK <-->|"T1-T10 calls / observations"| AGENT
    AGENT --> TRAJ["Trajectory T + diagnosis R_hat"]
    TRAJ --> SCORE["Outcome (A@k, TCR) +<br/>Process (alignment, IAC, RAR, ZTDR)"]
  end

The pipeline's self-correcting core is the Generator/Executor/Verifier loop in Phase 2: many faults are naturally masked by Kubernetes' own resilience (a taint gets bypassed by rescheduling, for instance), so the Verifier monitors telemetry after injection and, on a mismatch between intended and actual state, sends the Generator back to strengthen the injection (for example, increasing resource-stress intensity) until the fault reliably manifests.⁴ Only after verification does the State Snapshot module freeze the crime scene and pre-render the ten-tool response surface that becomes the deterministic replay layer for Phase 3.

How to run it against your own agent¶

The published harness targets CrewAI as the orchestration engine with Pydantic tool schemas and Langfuse tracing, evaluated in Python 3.10.¹⁴ The ten diagnostic tools an agent can call are fixed and typed (Table 3 in the paper): GetResources, DescribeResource, GetAppYAML, GetServiceDependencies, GetRecentLogs, CheckServiceConnectivity, GetClusterConfiguration, GetAlerts, GetErrorLogs, CheckNodeServiceStatus.¹⁵ Wiring in a new agent means implementing calls against this fixed toolset and letting the harness intercept each call against the frozen snapshot rather than a live API server; because every case is pre-rendered, there is no cluster to provision, and a full 452-case run is the "seconds on a standard laptop" claim the paper makes for democratized replay.⁶

Score honestly across both axes from Table 4: outcome (A@1, A@3, Task Completion Rate) and process (Exact/In-Order/Any-Order Match, Relevance, Coverage, Steps, Invalid Action Count, Mean Time to Identify, Redundant Action Rate, Zero-Tool Diagnosis Rate). A model that looks strong on A@1 alone can be masking a high ZTDR (Claude-4-Sonnet: A@1 0.5 but ZTDR 0.32, meaning a third of its episodes never touch a tool before answering) or a high IAC (Qwen3-14B: 0.40).⁹ Report all of them, or a single accuracy number will hide exactly the failure modes this benchmark was built to surface.

How to extend it with new fault types¶

Fault types enter through Phase 1's <P, A, S> formalization: Prerequisites (required pre-injection cluster state, e.g. a ResourceQuota object), Fault Artifact (the defective YAML or ChaosBlade injection rule), and Activation Sequence (the ordered steps to apply P then A).⁴ Extending the taxonomy means writing a new metadata entry in this shape and passing it through the paper's own triad of gates before it is trustworthy: LLM-as-Judge refinement of the semantic description, human cloud-native expert review of the causal logic, and runtime verification in a sandbox cluster confirming the <P,A,S> tuple reliably triggers the intended symptom.⁴ Skipping the runtime-verification step is the single most likely way a home-grown fault case silently fails to reproduce, since Kubernetes' self-healing behavior (pod rescheduling, automatic restarts) routinely masks naively-specified injections, which is exactly why the paper built a closed Generator/Executor/Verifier loop instead of a one-shot injection script.⁴

To add a new fault category beyond the paper's seven (Admission Control, Scheduling, Startup, Runtime, Service Routing, Performance, Infrastructure), define the symptom-to-verification-step inversion rule that produces T*: the paper's ground-truth synthesis is rule-based, deterministically mapping a known injected fault (e.g., "Added a Taint") to the verification action that reveals it (e.g., "Execute describe node tool"), structured so symptom discovery is a logical prerequisite to root-cause confirmation (Strict Causal Precedence) while tolerating interstitial exploratory actions (Exploratory Noise Tolerance).⁵ The executed code below models exactly this scoring semantics.

How to interpret and report scores honestly¶

Four honesty traps the paper's own data surfaces, worth checking before publishing a Cloud-OpsBench number:

Any-Order beats In-Order for every model in Table 4 (e.g., Qwen3-235B: 0.41 vs 0.38); this is not noise, it quantifies that agents find the right evidence but do not organize it into the linear deductive chain a human expert would.¹¹ Report both, not just whichever is higher.
Low RAR is not automatically good. GPT-4o's near-zero RAR (0.02) correlates with its lowest-tier A@1 (0.49); the paper interprets high RAR in the top performer (DeepSeek-V3.2, 0.11) as functional self-correction, not wasted motion.¹¹ A step-minimization objective would optimize for the wrong thing.
High Tool Relevance does not imply high accuracy. Qwen3-14B's Relevance (0.63) sits between GPT-4o's and GPT-5's, but its A@1 (0.34) is roughly half of GPT-5's (0.67); the bottleneck the paper identifies is information integration (turning a correct tool call's output into a causal chain), a capability Relevance does not measure.¹¹
Stratify by fault explicitness before averaging. Explicit faults (Startup, Runtime, with direct Kubernetes Events like OOMKilled) average A@1 above 0.65; implicit faults (Admission, Performance, requiring cross-layer causal inference) average below 0.36.¹² A single blended A@1 across all 452 cases obscures which failure classes an agent is actually reliable on.

Executed model: trajectory alignment and tool-usage metrics¶

The formulas below (Section 4.2.2) are the paper's own definitions for Exact/In-Order/Any-Order Match and Tool Relevance/Coverage, implemented against a synthetic Admission-Control ground truth modeled on the paper's Case #1 (Fig. 4: a Deployment/ReplicaSet whose Pod creation is rejected at admission time by an exhausted ResourceQuota -- GetResources{deployments} returns "frontend 0/1 ready" and DescribeResource on the ReplicaSet returns "forbidden: exceeded quota"; no Pod object is ever created, which is what makes this an Admission Control fault rather than the Scheduling category's "Pods stay Pending" symptom per Table 2). It is a model of the scoring semantics, not the CrewAI harness or the real 452-case dataset.

"""
Cloud-OpsBench process metrics (paper Section 4.2.2), executed and asserted.
Models: action matching (tool name + critical args must both match), the three
trajectory-alignment strictness levels (Exact / In-Order / Any-Order Match)
scored against a ground truth defined as a *minimal mandatory subsequence*
(Section 3.3.3, "Exploratory Noise Tolerance"), and Tool Relevance / Tool
Coverage computed over the *distinct* tool-call sets S_T the paper defines.
Pure stdlib. This is not the CrewAI evaluation harness; it is a faithful model
of the scoring semantics the paper specifies in prose and formula.
"""
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Action:
    tool: str
    args: tuple[tuple[str, str], ...]  # critical arguments only, order-independent

    def matches(self, other: "Action") -> bool:
        """Paper 4.2.2: a_i matches g_j iff both tool name AND critical
        arguments are identical -- same tool with a different target does
        not count (models the paper's 'schema hallucination' distinction)."""
        return self.tool == other.tool and self.args == other.args


def exact_match(agent: list[Action], gold: list[Action]) -> bool:
    """Section 4.2.2 (1): replicate the expert's exact sequence, no deviation."""
    if len(agent) != len(gold):
        return False
    return all(a.matches(g) for a, g in zip(agent, gold))


def in_order_match(agent: list[Action], gold: list[Action]) -> bool:
    """Section 4.2.2 (3): gold actions must appear as a subsequence of the
    agent's trajectory in the same relative order; interstitial/extra actions
    are allowed (Section 3.3.3's minimal mandatory subsequence definition)."""
    it = iter(agent)
    return all(any(a.matches(g) for a in it) for g in gold)


def any_order_match(agent: list[Action], gold: list[Action]) -> bool:
    """Section 4.2.2 (2): every gold action present somewhere; order and
    extra actions do not matter."""
    return all(any(a.matches(g) for a in agent) for g in gold)


def distinct_tools(traj: list[Action]) -> set[Action]:
    """S_T: the set of distinct tool calls in trajectory T (Section 4.2.2).
    Repeats of the same call collapse to one element, so redundant retries
    do not inflate Relevance/Coverage."""
    return set(traj)


def relevance(agent: list[Action], gold: list[Action]) -> float:
    """Relevance = |S_agent n S_gold| / |S_agent| (Section 4.2.2)."""
    s_agent, s_gold = distinct_tools(agent), distinct_tools(gold)
    if not s_agent:
        return 0.0
    return len(s_agent & s_gold) / len(s_agent)


def coverage(agent: list[Action], gold: list[Action]) -> float:
    """Coverage = |S_agent n S_gold| / |S_gold| (Section 4.2.2)."""
    s_agent, s_gold = distinct_tools(agent), distinct_tools(gold)
    if not s_gold:
        return 0.0
    return len(s_agent & s_gold) / len(s_gold)


# --- Ground truth for a synthetic Admission-Control fault, modeled on the
# paper's Case #1 (Fig. 4): a Deployment/ReplicaSet whose Pod creation is
# rejected at admission time by an exhausted ResourceQuota (no Pod object
# is ever created -- this is Admission Control, not the Scheduling
# category's "Pods stay Pending" symptom). Minimal mandatory subsequence
# T* per Section 3.3.3: symptom discovery must logically precede
# root-cause confirmation.
GOLD = [
    Action("GetResources", (("resource_type", "pods"),)),
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
    Action("GetResources", (("resource_type", "resourcequota"),)),
]

# 1) A high-fidelity agent trajectory: hits every gold step in order but
# interleaves redundant re-confirmation actions (paper's "redundancy
# paradox", RAR > 0 for the best-scoring model, DeepSeek-V3.2 in Table 4).
agent_thorough = [
    Action("GetResources", (("resource_type", "pods"),)),
    Action("GetResources", (("resource_type", "deployments"),)),          # extra
    Action("GetResources", (("resource_type", "pods"), ("label", "app=frontend"))),  # extra
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
    Action("DescribeResource", (("resource_type", "replicaset"), ("name", "frontend"))),  # extra
    Action("GetResources", (("resource_type", "resourcequota"),)),
]
assert exact_match(agent_thorough, GOLD) is False        # extra steps break exact match
assert in_order_match(agent_thorough, GOLD) is True       # gold subsequence intact, in order
assert any_order_match(agent_thorough, GOLD) is True
assert relevance(agent_thorough, GOLD) == 3 / 6           # 3 of 6 distinct calls are gold
assert coverage(agent_thorough, GOLD) == 3 / 3            # all 3 gold calls hit

# 2) Adversarial: a "lazy leap" agent that jumps straight to the root-cause
# check before establishing the symptom, then backfills. Same *set* of
# actions as the gold, so Any-Order Match still passes, but the causal
# precedence gold requires is violated, so In-Order Match must fail. This
# reproduces the paper's headline finding that Any-Order consistently beats
# In-Order (Table 4, e.g. Qwen3-235B 0.41 vs 0.38, Section 4.4).
agent_leap = [
    Action("GetResources", (("resource_type", "resourcequota"),)),        # jumps to root cause first
    Action("GetResources", (("resource_type", "pods"),)),
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
]
assert in_order_match(agent_leap, GOLD) is False
assert any_order_match(agent_leap, GOLD) is True
assert exact_match(agent_leap, GOLD) is False

# 3) Adversarial: schema hallucination / wrong target. Same tool name as the
# gold's root-cause check but a different critical argument (wrong
# namespace), so it must NOT count as a match even though the tool string is
# identical -- this is what the paper's Invalid Action Count / schema
# hallucination failure mode (Section 4.5, Fig. 4 Case #3) actually breaks.
agent_wrong_target = [
    Action("GetResources", (("resource_type", "pods"),)),
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
    Action("GetResources", (("resource_type", "resourcequota"), ("namespace", "staging"))),  # wrong ns
]
assert any_order_match(agent_wrong_target, GOLD) is False
assert coverage(agent_wrong_target, GOLD) == 2 / 3
assert relevance(agent_wrong_target, GOLD) == 2 / 3

# 4) Adversarial edge case: Zero-Tool Diagnosis (ZTDR, Section 4.2.2). The
# agent emits a final answer with no tool calls at all -- must not crash on
# an empty trajectory and must score as a total miss on every metric.
agent_ztdr: list[Action] = []
assert exact_match(agent_ztdr, GOLD) is False
assert in_order_match(agent_ztdr, GOLD) is False
assert any_order_match(agent_ztdr, GOLD) is False
assert relevance(agent_ztdr, GOLD) == 0.0
assert coverage(agent_ztdr, GOLD) == 0.0

# 5) Redundancy must not inflate Relevance/Coverage: repeating the same
# correct call 5 times collapses to one element of S_agent (Section 4.2.2
# defines S_T over distinct calls), so Relevance is unchanged versus a
# single clean hit, not diluted or amplified by retry count.
agent_repetitive = GOLD + [GOLD[-1]] * 4
assert distinct_tools(agent_repetitive) == distinct_tools(GOLD)
assert relevance(agent_repetitive, GOLD) == 1.0
assert coverage(agent_repetitive, GOLD) == 1.0
assert exact_match(agent_repetitive, GOLD) is False  # length differs (5 extra steps)

print("thorough  : exact=%s in_order=%s any_order=%s rel=%.2f cov=%.2f" % (
    exact_match(agent_thorough, GOLD), in_order_match(agent_thorough, GOLD),
    any_order_match(agent_thorough, GOLD), relevance(agent_thorough, GOLD), coverage(agent_thorough, GOLD)))
print("leap      : in_order=%s any_order=%s (Any-Order > In-Order reproduced)" % (
    in_order_match(agent_leap, GOLD), any_order_match(agent_leap, GOLD)))
print("wrong_tgt : any_order=%s rel=%.2f cov=%.2f (schema-mismatch action excluded)" % (
    any_order_match(agent_wrong_target, GOLD), relevance(agent_wrong_target, GOLD), coverage(agent_wrong_target, GOLD)))
print("ztdr      : exact=%s in_order=%s any_order=%s rel=%.2f cov=%.2f" % (
    exact_match(agent_ztdr, GOLD), in_order_match(agent_ztdr, GOLD), any_order_match(agent_ztdr, GOLD),
    relevance(agent_ztdr, GOLD), coverage(agent_ztdr, GOLD)))
print("repetitive: distinct_calls=%d rel=%.2f cov=%.2f (retries do not inflate score)" % (
    len(distinct_tools(agent_repetitive)), relevance(agent_repetitive, GOLD), coverage(agent_repetitive, GOLD)))
print("all trajectory-metric assertions passed")

Executed output:

thorough  : exact=False in_order=True any_order=True rel=0.50 cov=1.00
leap      : in_order=False any_order=True (Any-Order > In-Order reproduced)
wrong_tgt : any_order=False rel=0.67 cov=0.67 (schema-mismatch action excluded)
ztdr      : exact=False in_order=False any_order=False rel=0.00 cov=0.00
repetitive: distinct_calls=3 rel=1.00 cov=1.00 (retries do not inflate score)
all trajectory-metric assertions passed

Failure modes¶

Pitfall	Cause	Fix
Reporting a single A@1 as "the" score	Averages across explicit faults (A@1 > 0.65, e.g. OOMKilled) and implicit faults (A@1 < 0.36, e.g. Admission/Performance) mask a bimodal distribution.¹²	Break results out by fault category before comparing agents or claiming production readiness.
Optimizing an agent to minimize Steps or RAR	The paper's own data shows the highest-accuracy model (DeepSeek-V3.2) has the most steps (10.0) and highest RAR (0.11); GPT-4o's near-zero RAR (0.02) correlates with its weakest A@1 (0.49).¹¹	Treat redundant verification as a feature to preserve, not a cost to eliminate, unless MTTI is a hard product constraint.
Trusting Tool Relevance as a proxy for diagnostic quality	Qwen3-14B matches GPT-4o's Relevance (0.63) but trails badly on A@1 (0.34 vs 0.49); tool selection and information integration are separate capability layers.¹¹	Score outcome and process metrics together; never gate on Relevance/Coverage alone.
Assuming State Snapshot faithfully models live-cluster stochasticity	The paradigm freezes the crime scene by design (Threats to Validity, Construct Validity); it explicitly cannot capture real-time state transitions or trial-and-error remediation.¹³	Pair with a live-environment benchmark like AIOpsLab before shipping an agent that must also act on a cluster, not just diagnose it.
Skipping runtime verification when authoring new fault cases	Kubernetes' self-healing (pod rescheduling, automatic restarts) silently masks naive injections; the paper's own Generator/Executor/Verifier loop exists because of this.⁴	Run the sandbox-cluster verification step (the paper's third gate) before committing a new `<P,A,S>` tuple to a fault knowledge base.
Citing the 40-fault-type taxonomy as covering "cloud failures" generally	Every fault type and tool is Kubernetes-specific (Table 2, Table 3); the authors flag monolithic/serverless transfer as unverified.¹³	Scope claims to Kubernetes-native RCA; validate separately before extending to other platforms.

References¶

Wang, Yu, Huang, Wang, Huang, Chen, Lyu, Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems (arXiv 2603.00468): https://arxiv.org/abs/2603.00468
PDF: https://arxiv.org/pdf/2603.00468
Code and dataset (452 fault cases): https://github.com/LLM4Ops/Cloud-OpsBench
AIOpsLab (MLSys 2025), the live-environment benchmark Cloud-OpsBench positions itself against: https://arxiv.org/abs/2501.06706
OpenRCA (ICLR 2025), the static-artifact benchmark with generative (code-synthesis) tool use: https://openreview.net/forum?id=M4qNIzQYpd
Google Online Boutique microservices demo (the workload used as the testbed): https://github.com/GoogleCloudPlatform/microservices-demo
ChaosBlade fault-injection tool: https://chaosblade.io/

Section 3.1, "Agentic RCA Task Formulation": ground truth tuple R* = <S, C, R> (Stage, Component, Root Cause); task modeled as f: <A_alert, E_snapshot> -> <T, R_hat> (Eq. 1); trajectory T = [(t_1,a_1,o_1), ..., (t_n,a_n,o_n)] of thought/action/observation steps. ↩
Section 3.3.3, "2. State Snapshot": exhaustive parameter sweep against tools T1-T10 (Table 3); "each fault case generates 487 distinct tool invocations" on average; valid queries hit deterministic entries, invalid queries map to realistic Not Found errors; reasoning search space stated as 487^N with N approx 10 required interdependent steps, argued to make random guessing statistically impossible. ↩↩
Section 3.3.1, "Benchmark Testbed": Huawei Cloud ECS, Kubernetes v1.31 on 4 instances, Prometheus + Istio observability, Google Online Boutique (11 microservices, gRPC) driven by Locust, ChaosBlade for infrastructure-level perturbation; Table 2 taxonomy: 7 fault categories, 40 fault types, 452 total cases (Admission Control 58, Scheduling 164, Startup 62, Runtime 45, Service Routing 54, Performance 21, Infrastructure 48). ↩
Section 3.3.2 (Phase 1) and 3.3.3 (Phase 2): fault metadata as <Semantic Metadata, <P,A,S>> (Prerequisites, Fault Artifact, Activation Sequence); validation triad LLM-as-Judge (Gemini 3 Pro) -> human expert -> sandbox runtime verification; Generator/Executor/Verifier closed loop, Verifier re-triggers Generator on injection masking (e.g. taint bypassed by rescheduling). ↩↩↩↩↩↩
Section 3.3.3, "3. Ground Truth Annotation": G* = <R*, T*>; T synthesized by rule-based inversion of the injection logic; two design principles named verbatim: Strict Causal Precedence (Anti-Guessing) and Exploratory Noise Tolerance (Pro-Redundancy), the latter defining T as "a minimal mandatory subsequence rather than a rigid script." ↩
Section 5.1, "Significance and Impact": "Democratizing Research via Zero-Cost Replay" paragraph states prior live-cluster benchmarks require "often hours per run" while Cloud-OpsBench "enables a full-scale benchmark run in seconds on a standard laptop"; Section 5.2 frames the benchmark as a data engine for SFT bootstrapping via harvested trajectories. ↩↩↩
Section 2.3, "Gap 2: The Reproducibility Challenge in Dynamic Environments": network-jitter example (a 500ms delay injection triggering cascading timeouts in one run, seamless retry in another) used to argue live A/B testing across models is invalid. ↩
Section 4.2, "Evaluation Metrics" and 4.2.2 "Process-based Metrics": A@k formula (Section 4.2.1); trajectory alignment three strictness levels (Exact/Any-Order/In-Order Match) with formal match condition (tool name + critical arguments identical); Relevance and Coverage formulas over distinct tool-call sets S_T. ↩
Table 4, "Performance comparison of LLMs in Agentic RCA on outcome and process metrics": DeepSeek-V3.2 A@1 0.73, A@3 0.79, Steps 10.0, Coverage 0.88, RAR 0.11; GPT-4o A@1 0.49, MTTI 23.27, Steps 5.67, RAR 0.02; GPT-5 A@1 0.67, Relevance 0.65, Steps 5.57; Qwen3-14B Relevance 0.63, IAC 0.40, A@1 0.34; Qwen3-8B A@1 0.21; Claude-4-Sonnet A@1 0.5, ZTDR 0.32, Steps 4.25; discussed in Sections 4.3-4.5. ↩↩↩
Table 5, "Performance comparison of knowledge-enhanced agents" and Section 4.6 "RQ4: Knowledge Efficacy": GPT-4o Base A@1 0.49 -> ICL 0.7 vs RAG 0.61; Qwen3-14B Base A@1 0.34 -> ICL 0.71 (IAC 0.40 -> 0.29) vs RAG 0.5; Qwen3-235B CoT degrades A@1 from 0.50 to 0.47. ↩
Section 4.4, "RQ2: RCA Process Alignment": Qwen3-235B Any-Order 0.41 vs In-Order 0.38; DeepSeek-V3.2 RAR 0.11 termed a "redundancy paradox" and functional self-correction; GPT-4o RAR 0.02 with lower accuracy; DeepSeek-V3.2 called "Coverage-Centric," GPT-5 called "Relevance-Centric"; Qwen3-14B Relevance 0.63 vs A@1 0.34 gap attributed to "information integration," not tool selection. ↩↩↩↩↩
Section 4.3, "RQ1: RCA Outcome Effectiveness," "Performance stratification is determined by symptom explicitness and observability": explicit faults (Startup, Runtime) "Avg A@1 > 0.65"; implicit faults (Admission, Performance) "Avg A@1 < 0.36." ↩↩
Section 5.3, "Threats to Validity": External Validity notes Kubernetes-specific toolsets may not generalize to monolithic/serverless platforms; Construct Validity states the State Snapshot paradigm "restricts the agent's ability to observe real-time state transitions" and explicitly scopes the benchmark to reasoning over existing evidence, "distinguishing itself from downstream tasks that require trial-and-error remediation or active environmental intervention." ↩↩↩↩
Section 4.1, "Experimental Setup": Python 3.10, CrewAI orchestration, Pydantic tool schemas, Langfuse observability; seven models in two tiers (LLM tier: GPT-5, GPT-4o, Claude-4-Sonnet, DeepSeek-V3.2, Qwen3-235B; SLM tier under 20B: Qwen3-14B, Qwen3-8B). ↩
Table 3, "Description of specialized diagnostic tools in Cloud-OpsBench": T1 GetResources, T2 DescribeResource, T3 GetAppYAML, T4 GetServiceDependencies, T5 GetRecentLogs, T6 CheckServiceConnectivity, T7 GetClusterConfiguration, T8 GetAlerts, T9 GetErrorLogs, T10 CheckNodeServiceStatus. ↩