Markdown

Cloud-OpsBench: a reproducible State Snapshot benchmark for agentic RCA¶

Scope: Cloud-OpsBench (arXiv 2603.00468, CUHK/Sun Yat-sen University), a 452-case, 40-fault-type benchmark for agentic Root Cause Analysis (RCA) on Kubernetes that replaces live-cluster evaluation with a deterministic "State Snapshot" digital twin, its three-phase generation pipeline (fault-knowledge-base construction, closed-loop fault injection with Generator/Executor/Verifier agents, and snapshot freezing), its process-centric metrics (trajectory alignment, tool relevance/coverage, invalid-action and zero-tool-diagnosis rates), and the seven-model evaluation results. The live-cluster counterpart this benchmark is positioned against is AIOpsLab; the field view of running agentic operations on a cluster (task taxonomy, guardrails, evaluation discipline) is agentic AIOps; a worked production multi-agent RCA system this benchmark's findings would gate is OpsAgent.

Independently verified/executed here: the repository at https://github.com/LLM4Ops/Cloud-OpsBench, pinned at commit 0faa0b8d53d16b153b7300c03f5a29791aea9cb9 (2026-07-16), was cloned and installed in a clean virtualenv (Python 3.12.3, pydantic==2.13.4, openai==2.45.0, PyYAML==6.0.3) and its real, unmodified harness code was run end to end against the real, on-disk benchmark case benchmark/boutique/performance/22: tools.definition.create_k8s_tools built the real Pydantic-typed tool set from the real snapshot files, runtime.agent_runtime.AgentRuntime drove a full ReAct loop through the real PromptBuilder, ToolExecutor (calling the real KubernetesTools methods, not a mock), OutputParser, and TraceLogger, and diagnostic_evidence.evaluator.evaluate_trajectory scored the resulting trace against the real process-label/boutique/performance/22/milestone.json. Only the LLM call itself was replaced, with a scripted StubAgentAdapter returning a fixed transcript, because this sandbox has no LLM API key; the complete driver (not an excerpt) and its real output are reproduced below, together with three adversarial harness-boundary cases (unknown tool name, a tool call missing a required argument, and malformed Action Input JSON), also run through the complete real code, not just pasted output. Re-review caught two errors in an earlier revision, both corrected below with the real driver script that surfaces them: the harness-wiring code itself was missing (only the StubAgentAdapter class was shown, with the actual AgentRuntime construction and run_case call asserted but not reproduced), and the claim that scoring needs an "unshipped workaround" for a KeyError: 'description' was wrong, the repository's own cloudops_agent.evaluation.load_process_annotation() already handles it; there is no repository defect to file. A further re-verification pass on 2026-07-17 cloned the pinned commit again from scratch and reran both driver scripts unmodified: the scoring result matched exactly, but the happy-path transcript pasted in an earlier revision was missing two lines of real stdout (GetAlerts:{} and GetClusterConfiguration:{}), printed unconditionally by the real, unmodified KubernetesTools methods before every cache lookup; the transcript below is corrected to include them. That same pass also found the pin is already one commit behind origin/main, which renamed the entire diagnostic_evidence/ package this page imports from; the new "Maintenance, CI, and version rollback" section documents this as a worked example, since the repository has no CI, no test suite, and no tags of its own to catch it for a reader. Separately, the Python trajectory-alignment model further down (Exact/In-Order/Any-Order Match, Tool Relevance, Tool Coverage) is executed and asserted against synthetic cases matching the paper's Section 4.2.2 formulas; it is a model of that scoring semantics, not the harness. All numbers in "Why use it" and the results discussion are quoted from the paper's tables and sections, each with a footnote pointing at the specific table/section; the paper's headline results were not reproduced by running the benchmark, and, as the drift note below documents, the live repository has moved past what the paper describes.

What it is¶

Cloud-OpsBench is a benchmark that formalizes agentic RCA as a trajectory-based decision process, f: <A_alert, E_snapshot> -> <T, R_hat>: an agent receives a natural-language alert and a cached system snapshot, then produces both a diagnostic trajectory (the sequence of thought/action/observation steps it took) and a predicted root-cause tuple, and both are scored.¹ Its central design contribution is the State Snapshot Paradigm: instead of deploying a live Kubernetes cluster and letting an agent query a volatile API server, the benchmark pre-executes an exhaustive parameter sweep of its ten diagnostic tools against a frozen fault state, so every plausible kubectl-style query the agent could issue resolves to a pre-recorded, deterministic answer, on average 487 distinct tool invocations get pre-computed per fault case.² Interactivity is reconstructed through a mocked operational interface: valid queries hit a deterministic entry, invalid queries (nonexistent pod names, wrong namespaces) map to realistic "Not Found" errors, so the agent still has to generate syntactically correct, semantically targeted commands even though nothing it does can perturb a real cluster.²

The dataset itself comprises 452 fault cases across 40 fault types organized into 7 categories (Admission Control, Scheduling, Startup, Runtime, Service Routing, Performance, Infrastructure) spanning the full Kubernetes stack, built on a Google Online Boutique deployment (11 microservices) on a 4-node Kubernetes v1.31 cluster with Prometheus and Istio for telemetry.³ Each case is generated by a three-agent pipeline (Generator, Executor, Verifier) that closes the loop between abstract fault knowledge and a physically-verified injection, then annotates ground truth as <R*, T*>: a structured root-cause tuple and a canonical diagnostic path synthesized by inverting the injection logic.⁴

Repository drift from the published paper (checked 2026-07-16)¶

The paper (arXiv 2603.00468) and the live main branch of the repository no longer describe the same artifact. Anyone building against this benchmark today needs the repository's current shape, not the paper's, for anything code-facing:

Dimension	Paper (arXiv 2603.00468)	Repository at `0faa0b8d5` (2026-07-16)
Orchestration engine	CrewAI, Pydantic tool schemas, Langfuse tracing¹⁴	A custom single-agent ReAct runtime (`cloudops_agent/runtime/`); no CrewAI or Langfuse dependency anywhere in the tree.
Systems covered	One (Online Boutique)	Two: Online Boutique (550 cases) and Train-Ticket (204 cases).
Fault types / cases	40 fault types, 452 cases³	57 fault types, 754 cases, including a new "Application Code Defect" category with `ListCodeFiles`/`GetSourceCode` tools the paper never describes.
Diagnostic tools	10 (Table 3)¹⁵	12 for Online Boutique (10 shared plus `ListCodeFiles`, `GetSourceCode`), 10 for Train-Ticket.
Process-evaluation design	Trajectory alignment (Exact/In-Order/Any-Order Match) plus Relevance/Coverage over the full trajectory⁸	Milestone-based process annotations (`process-label/*/milestone.json`), added 2026-07-16 per the README changelog, scored by `diagnostic_evidence/evaluator.py` as Milestone Coverage (MC), Evidence-Order Consistency (EOC), Evidence Closure Rate (ECR), and Evidence Efficiency (EE); the paper's Exact/In-Order/Any-Order Match metrics do not appear in the current evaluator at all.
Reported metrics	A@1, A@3, TCR, IAC, MTTI, RAR, ZTDR (Table 4)⁹	CA, FA, JRA, MC, EOC, ECR, EE, Steps, RAR (README's own results tables, by system)
Models evaluated	7 (GPT-5, GPT-4o, Claude-4-Sonnet, DeepSeek-V3.2, Qwen3-235B/14B/8B)	10 per system (GPT-5, Claude-Sonnet-4, Gemini-2.5-Pro/Flash, Qwen3.5-Plus/27B, Qwen3-235B-A22B/14B/8B, DeepSeek-V4-Flash); no GPT-4o or Claude-4-Sonnet row remains.

None of the paper citations elsewhere on this page are wrong as citations; they accurately quote the paper as published. The point is narrower and more important operationally: do not point a reader at the paper's Table 3/Table 4 numbers or CrewAI setup and expect them to match what git clone gets today. Everything under "How to run it" below targets the repository as it actually is at the pinned commit.

Why use it¶

Zero-cost, fully reproducible replay. By decoupling state storage from runtime execution, a full benchmark run drops from the "often hours per run" the paper attributes to live-cluster evaluation to seconds on a standard laptop, removing the industrial-cluster barrier to entry the paper explicitly targets.⁶
It scores the investigation, not just the guess. Trajectory alignment (Exact/In-Order/Any-Order Match) and tool-usage metrics (Relevance, Coverage) catch an agent that is "right for the wrong reasons," which pure outcome accuracy (A@k) cannot distinguish from systematic reasoning.⁸
The results overturn the "shorter is better" intuition. DeepSeek-V3.2 posts the best outcome accuracy (A@1 0.73, A@3 0.79) with the longest trajectories (10.0 steps) and highest Coverage (0.88), while GPT-4o converges fastest (MTTI 23.27s, 5.67 steps) but scores lowest A@1 (0.49) among the six evaluated frontier/near-frontier models; the paper reads this as a "redundancy paradox" where DeepSeek-V3.2's 11% Redundant Action Rate functions as self-correction, not waste.⁹
It isolates a real bottleneck in small models. Qwen3-14B matches GPT-4o's Tool Relevance (0.63 vs 0.63) and nearly matches GPT-5's (0.65), meaning it knows which tool to call, but its Invalid Action Count (0.40) runs about 10x GPT-5's (0.04), and its A@1 (0.34) trails GPT-5 (0.67) by 33 points; the gap is malformed API calls and hallucinated parameters, not diagnostic judgment.⁹
In-context demonstrations, not manuals, close the gap. Retrieving 3 historical diagnostic traces (ICL) lifted Qwen3-14B's A@1 from 0.34 to 0.71 (matching unprompted GPT-5-tier performance) by cutting its Invalid Action Count from 0.40 to 0.29, while RAG over Kubernetes documentation only reached 0.50; procedural examples fix syntax, declarative docs do not.¹⁰

When to use it (and when not)¶

Use it to A/B test candidate RCA agents (model swaps, prompting strategies, tool-schema changes) under bit-for-bit identical fault conditions, something a live cluster cannot guarantee because of network jitter and scheduling nondeterminism.⁷
Use it as a pre-deployment gate the way AIOpsLab is used: check Tool Relevance/Coverage and Invalid Action Count before trusting an agent with real tool access, since the paper shows outcome accuracy alone hides syntactic fragility.
Use it as a training-data engine: the same pre-computed trajectories that power evaluation are the "harvested" demonstrations the paper uses for ICL and proposes for SFT bootstrapping of small models.⁶
Do not treat it as a remediation or intervention benchmark. The paper explicitly scopes RCA to reasoning over frozen, post-mortem evidence; it does not exercise mitigation actions or trial-and-error environmental intervention, unlike AIOpsLab's mitigation task level.¹³
Do not assume the fault taxonomy transfers past Kubernetes. The tools (kubectl-shaped) and fault categories are Kubernetes-specific; the authors themselves flag that monolithic or serverless architectures are out of scope, though they argue the underlying reasoning patterns (resource contention, cascading failure) generalize.¹³
Do not cite the seven-model leaderboard as current capability. It is a point-in-time snapshot (GPT-5/GPT-4o/Claude-4-Sonnet/DeepSeek-V3.2/Qwen3-235B/Qwen3-14B/Qwen3-8B as of the paper's writing); re-run it yourself before making a procurement or architecture decision.

Architecture¶

flowchart TB
  subgraph P1["Phase 1: Fault Knowledge Base Construction"]
    SRC["K8s docs, Stack Overflow,<br/>academic papers"] --> GEM["Gemini 3 Pro: extract<br/>structured metadata"]
    GEM --> DRAFT["Draft fault metadata:<br/>semantic + <P,A,S> tuple"]
    DRAFT --> JUDGE["LLM-as-Judge refine"]
    JUDGE --> EXPERT["Human expert review"]
    EXPERT --> KB["Validated Fault<br/>Knowledge Base<br/>(40 fault types)"]
  end

  subgraph P2["Phase 2: Automatic Fault Case Generation (closed loop)"]
    KB --> GENA["Generator Agent:<br/>CoT -> fault plan M=<P,A,S>"]
    CLUSTER["Target Kubernetes cluster<br/>(Online Boutique, 4 nodes)"] --> GENA
    GENA --> EXECA["Executor Agent:<br/>kubectl / ChaosBlade injection"]
    EXECA --> VERA["Verifier Agent:<br/>telemetry check"]
    VERA -->|"fault masked/not triggered"| GENA
    VERA -->|"verified"| SNAP["State Snapshot:<br/>exhaustive T1-T10 sweep,<br/>~487 tool calls/case"]
    SNAP --> GT["Ground truth <R*,T*>:<br/>injection logic inverted"]
  end

  subgraph P3["Phase 3: Benchmarking"]
    GT --> CASES["452 fault cases"]
    SNAP --> MOCK["Mocked tool interface<br/>(deterministic replay)"]
    CASES --> AGENT["Agent under test"]
    MOCK <-->|"T1-T10 calls / observations"| AGENT
    AGENT --> TRAJ["Trajectory T + diagnosis R_hat"]
    TRAJ --> SCORE["Outcome (A@k, TCR) +<br/>Process (alignment, IAC, RAR, ZTDR)"]
  end

The pipeline's self-correcting core is the Generator/Executor/Verifier loop in Phase 2: many faults are naturally masked by Kubernetes' own resilience (a taint gets bypassed by rescheduling, for instance), so the Verifier monitors telemetry after injection and, on a mismatch between intended and actual state, sends the Generator back to strengthen the injection (for example, increasing resource-stress intensity) until the fault reliably manifests.⁴ Only after verification does the State Snapshot module freeze the crime scene and pre-render the ten-tool response surface that becomes the deterministic replay layer for Phase 3.

How to run it against your own agent¶

This section documents the repository's actual current runtime (the custom ReAct loop, not the paper's CrewAI setup) with commands and code executed in the course of writing this page.

1. Pinned checkout and environment¶

git clone https://github.com/LLM4Ops/Cloud-OpsBench.git
cd Cloud-OpsBench
git checkout 0faa0b8d53d16b153b7300c03f5a29791aea9cb9

python3 -m venv .venv
source .venv/bin/activate
pip install "pydantic==2.13.4" "openai==2.45.0" "PyYAML==6.0.3"

The repository ships no requirements.txt or pyproject.toml; the three packages above are every third-party import under cloudops_agent/ and diagnostic_evidence/ (confirmed by grepping every import/from line in both packages), pinned to the versions actually used to run the commands below on Python 3.12.3. There is also no LICENSE file in the repository as of this commit; check with the authors before redistributing or productionizing code from it.

2. Case and result layout¶

Cloud-OpsBench/
├── benchmark/<system>/<fault_category>/<case_id>/
│   ├── metadata.json     # {namespace, query, difficulty, result: {fault_taxonomy, fault_object, root_cause}}
│   ├── tool_cache.json   # pre-rendered deterministic tool responses, keyed by "ToolName:{json-args}"
│   ├── code/             # trimmed business-logic source (Online Boutique only)
│   └── raw_data/         # alert.json, k8s_states.json, logs.json, metrics.csv (not all present per case)
├── process-label/<system>/<fault_category>/<case_id>/milestone.json   # ground-truth diagnostic milestones
├── golden-trajectory/<system>/<fault_category>/<case_id>/{path1,path2}.json  # auxiliary expert traces (unused by current scoring)
└── cloudops_agent/
    ├── configs/model_configs.yaml   # {model: {model, provider, api_base, api_key, temperature, max_tokens, timeout}, diagnosis: {max_iterations, system, fault_category, dataset_root, save_root, case_name?}}
    ├── run.py           # entry point: reads model_configs.yaml, runs every case under system/fault_category
    └── evaluation.py    # entry point: scores the trajectories run.py produced

system is "boutique" or "trainticket"; fault_category is one of admission, scheduling, startup, runtime, service, performance, infrastructure, or (Online Boutique only) codedefect. Result trajectories land at <save_root>/<model_name>/<fault_category>/<case_id>/<case_id>.json.

3. Baseline run against a real OpenAI-compatible endpoint¶

With a working api_base/api_key in cloudops_agent/configs/model_configs.yaml (provider: openai_compatible, so any OpenAI-compatible server, self-hosted or vendor, works), the published entry points are:

cd cloudops_agent
python run.py          # writes <save_root>/<model>/<fault_category>/<case_id>/<case_id>.json per case
python evaluation.py   # reads the same config, scores the trajectories run.py just wrote

This sandbox has no LLM API key, so run.py/evaluation.py were not executed end to end here; what follows instead replaces only ModelRunner.generate() and drives every other real module, so the tool-execution, parsing, and scoring code paths are the same code run.py calls.

4. Typed tool schemas and the agent-adapter contract¶

Each of the 12 (10 for Train-Ticket) diagnostic tools is a SimpleTool wrapping a Pydantic args_schema, built by tools.definition.create_k8s_tools(case_path, system, fault_category):

class GetResourcesInput(BaseModel):
    resource_type: str = Field(description="REQUIRED. e.g. 'pods', 'services', 'deployments', 'nodes'.")
    namespace: Optional[str] = Field(default=None, description="Required for namespaced resources.")
    name: Optional[str] = Field(default=None, description="Optional: return a single named resource.")
    show_labels: bool = Field(default=False)
    output_wide: bool = Field(default=False)
    label_selector: Optional[str] = Field(default=None)

tools.registry.render_tools_description reflects these Pydantic field definitions (name, required/optional, type, description) straight into the prompt text, so the schema shown to the model always matches the schema tools.adapters.call_tool validates against at execution time. The only interface a custom agent must implement to replace the shipped model call is ModelRunner.generate(prompt: str) -> {"text": str, "latency": float, "input_tokens": int, "output_tokens": int, "raw_response": Any}; AgentRuntime calls nothing else on it.

5. Executed: a scripted adapter through the real harness (no LLM, no mocks elsewhere)¶

The following is the complete driver, written and run in this sandbox against the real, unmodified cloudops_agent/diagnostic_evidence code and the real, on-disk case benchmark/boutique/performance/22, not an excerpt. Only StubAgentAdapter.generate() is scripted; every other call (create_k8s_tools, build_tool_registry, PromptBuilder, ToolExecutor, which calls the real KubernetesTools methods against the real snapshot files, OutputParser, TraceLogger, AgentRuntime.run_case, load_process_annotation, agent_trajectory, and diagnostic_evidence.evaluator.evaluate_trajectory) is the harness's own code, imported and called unmodified (caught on re-review: an earlier revision of this page showed only the StubAgentAdapter class and asserted the rest was "wired... exactly as run.py does" without showing that wiring; a reader had no way to reproduce it from the page alone):

# smoke_driver.py, run from the repo root: python3 smoke_driver.py
from __future__ import annotations
import json, sys
from pathlib import Path

sys.path.insert(0, str(Path("cloudops_agent").resolve()))
from prompts.RCA_candidate import agent_prompt, build_expected_output
from tools.definition import create_k8s_tools
from tools.registry import build_tool_registry, render_tools_description
from runtime.state import init_case_state
from runtime.prompt_builder import PromptBuilder
from runtime.output_parser import OutputParser
from runtime.tool_executor import ToolExecutor
from runtime.logger import TraceLogger
from runtime.agent_runtime import AgentRuntime

sys.path.insert(0, str(Path(".").resolve()))
from cloudops_agent.evaluation import load_process_annotation, agent_trajectory
from diagnostic_evidence.evaluator import evaluate_trajectory


class StubAgentAdapter:
    """Drop-in replacement for runtime.model_runner.ModelRunner: implements only
    .generate(prompt) -> dict, the sole interface AgentRuntime depends on."""

    def __init__(self) -> None:
        self._step = 0
        self._script = [
            'Thought: Check for active alerts first.\nAction: GetAlerts\nAction Input: {}',
            'Thought: Correlate against cluster-wide node health.\nAction: GetClusterConfiguration\nAction Input: {}',
            (
                'Thought: worker-02 shows elevated network latency; alerts and cluster '
                'configuration both point at a network-layer fault on that node.\n'
                '{\n  "key_evidence_summary": "GetAlerts reported node-level network latency; '
                'GetClusterConfiguration confirmed worker-02 network condition degraded.",\n'
                '  "top_3_predictions": [\n'
                '    {"rank": 1, "fault_object": "node/worker-02", "root_cause": "node_network_delay"},\n'
                '    {"rank": 2, "fault_object": "node/worker-02", "root_cause": "node_network_packet_loss"},\n'
                '    {"rank": 3, "fault_object": "app/frontend", "root_cause": "pod_network_delay"}\n'
                '  ]\n}'
            ),
        ]

    def generate(self, prompt: str) -> dict:
        text = self._script[min(self._step, len(self._script) - 1)]
        self._step += 1
        return {"text": text, "latency": 0.0, "input_tokens": len(prompt.split()),
                "output_tokens": len(text.split()), "raw_response": None}


CASE_PATH = Path("benchmark/boutique/performance/22").resolve()
metadata = json.loads((CASE_PATH / "metadata.json").read_text())
namespace = metadata.get("namespace", "boutique")
query = metadata.get("query", "")
full_question = (
    f"The Kubernetes environment in namespace `{namespace}` is experiencing a fault. "
    f"A high-level symptom has been reported: '{query}'. Diagnose the root cause of this incident."
)

tools_list = create_k8s_tools(str(CASE_PATH), system="boutique", fault_category="performance")
tool_registry = build_tool_registry(tools_list)
prompt_builder = PromptBuilder(
    tools_description=render_tools_description(tool_registry),
    backstory_prompt=agent_prompt, expected_output=build_expected_output("boutique"),
)
trace_logger = TraceLogger(trace_dir="/tmp/cloudopsbench-smoke-trace")
runtime = AgentRuntime(
    prompt_builder=prompt_builder, model_runner=StubAgentAdapter(),
    output_parser=OutputParser(), tool_executor=ToolExecutor(tool_registry=tool_registry),
    trace_logger=trace_logger,
)

state = init_case_state(case_id="22", system_name="performance", question=full_question,
                         case_path=str(CASE_PATH), max_steps=10,
                         metadata={"namespace": namespace, "query": query,
                                   "result": metadata.get("result", "")})
final_state = runtime.run_case(state)
trace_path = trace_logger.get_trace_path(final_state)
print(json.dumps({"trace_path": str(trace_path), "finished": final_state.finished,
                   "stop_reason": final_state.stop_reason, "steps": len(final_state.history),
                   "final_answer_raw": final_state.final_answer}, indent=2))

# Real scoring path: cloudops_agent.evaluation.load_process_annotation, not
# diagnostic_evidence.schema.CaseAnnotation.from_dict called directly. See below.
annotation = load_process_annotation(Path("process-label/boutique"), "performance/22")
trace_data = json.loads(Path(trace_path).read_text())
result = evaluate_trajectory(annotation, agent_trajectory(trace_data))
print(json.dumps({"process_complete": result.process_complete,
                   "milestone_coverage": result.milestone_coverage,
                   "evidence_order_coverage": result.evidence_order_coverage,
                   "evidence_efficiency": result.evidence_efficiency,
                   "established_milestones": result.established_ids}, indent=2))

Real, unmodified output from python3 smoke_driver.py, re-executed on 2026-07-17 against a fresh clone at the same pinned commit to confirm this transcript is still reproducible:

GetAlerts:{}
GetClusterConfiguration:{}
{
  "trace_path": "/tmp/cloudopsbench-smoke-trace/22.json",
  "finished": true,
  "stop_reason": "final_answer",
  "steps": 3,
  "final_answer_raw": "{\"key_evidence_summary\": \"GetAlerts reported node-level network latency; GetClusterConfiguration confirmed worker-02 network condition degraded.\", \"top_3_predictions\": [{\"rank\": 1, \"fault_object\": \"node/worker-02\", \"root_cause\": \"node_network_delay\"}, {\"rank\": 2, \"fault_object\": \"node/worker-02\", \"root_cause\": \"node_network_packet_loss\"}, {\"rank\": 3, \"fault_object\": \"app/frontend\", \"root_cause\": \"pod_network_delay\"}]}"
}
{
  "process_complete": true,
  "milestone_coverage": 1.0,
  "evidence_order_coverage": 1.0,
  "evidence_efficiency": 1.0,
  "established_milestones": [
    "M1",
    "M2"
  ]
}

The established_milestones list above spans four lines, not one: the driver's own print(json.dumps(..., indent=2)) pretty-prints every nested list the same way it pretty-prints the surrounding dict, and an earlier revision of this page's transcript showed it compact on one line, which is not what that call produces. Corrected to the literal captured stdout after re-running the exact script above unmodified.

final_state.final_answer is the raw JSON string the model emitted, not a parsed dict; an earlier revision of this page showed "final_prediction": {"rank": 1, ...} as if the harness parsed it down to the top prediction, which is not what AgentRuntime actually returns. Corrected to the real field name (final_answer_raw) and the real, unparsed value.

The two GetAlerts:{} / GetClusterConfiguration:{} lines at the top are not a driver artifact: every method on the real, unmodified KubernetesTools class in cloudops_agent/tools/implement.py (for example GetAppYAML, GetServiceDependencies, and by the same pattern GetAlerts, GetClusterConfiguration) does an unconditional print(command_key) immediately before its tool_cache lookup, so every tool call the agent makes prints its cache key to stdout. An earlier revision of this page's transcript omitted these two lines; re-running the exact script above against a clean clone reproduced them, so they are restored here rather than treated as noise.

The "unshipped workaround" claim in an earlier revision of this page was wrong, caught on re-review. That revision called diagnostic_evidence.schema.CaseAnnotation.from_dict directly on the raw milestone.json dict, hit KeyError: 'description' (real: process-label/*/milestone.json files omit that field), patched it with a hand-rolled milestone.setdefault(...) before from_dict, and then asserted this workaround "is not part of the shipped code; file this against the repository." That is incorrect. The repository's own cloudops_agent/evaluation.py already ships load_process_annotation(), which does exactly this defaulting (milestone.setdefault("description", ""); milestone.setdefault("role", "")) before calling CaseAnnotation.from_dict, specifically to handle this file shape. The driver above uses that real function; there is no repository defect to file, and no custom workaround is needed. The defect was in the earlier revision calling the lower-level schema method directly instead of the harness's own intended entry point, not in the repository.

Three adversarial cases were run against the same real ToolExecutor/OutputParser to confirm the harness fails closed rather than crashing or silently miscounting; this is the complete driver, not an excerpt:

# adversarial_driver.py, run from the repo root: python3 adversarial_driver.py
from pathlib import Path
import json, sys
sys.path.insert(0, str(Path("cloudops_agent").resolve()))
from tools.definition import create_k8s_tools
from tools.registry import build_tool_registry
from runtime.output_parser import OutputParser
from runtime.tool_executor import ToolExecutor

CASE_PATH = Path("benchmark/boutique/performance/22").resolve()
tools_list = create_k8s_tools(str(CASE_PATH), system="boutique", fault_category="performance")
tool_registry = build_tool_registry(tools_list)
tool_executor = ToolExecutor(tool_registry=tool_registry)
output_parser = OutputParser()

unknown_result = tool_executor.execute("GetNonexistentTool", {})
bad_args_result = tool_executor.execute("DescribeResource", {"resource_type": "deployments"})
malformed_parsed = output_parser.parse('Thought: checking\nAction: GetAlerts\nAction Input: {"broken')

print(json.dumps({
    "unknown_tool_error": unknown_result.get("error"),
    "bad_args_success": bad_args_result.get("success"),
    "malformed_output_type": malformed_parsed.get("type"),
    "malformed_output_error": malformed_parsed.get("error"),
}, indent=2))

{
  "unknown_tool_error": "Unknown tool: GetNonexistentTool",
  "bad_args_success": false,
  "malformed_output_type": "invalid",
  "malformed_output_error": "Tool parse error: Found Action but missing or malformed Action Input. | Final JSON parse error: Final JSON parse failed: Expecting value: line 1 column 1 (char 0)"
}

The driver above prints only this JSON blob; an earlier revision of this page's transcript prepended a === Adversarial: ... === heading line that the script never actually prints (no print() call for it exists in the driver shown), so it is removed here to match the real, unmodified script's real stdout.

An unknown tool name returns a structured error rather than raising; a DescribeResource call missing its required name argument fails rather than executing with a null target; and truncated Action Input JSON is reported as type: "invalid" by OutputParser rather than crashing the run loop, which is what lets AgentRuntime.run_case continue to max_steps instead of aborting the whole case on one malformed generation.

6. Scoring with the harness's own current metrics¶

The evaluator this repository ships today is milestone-based (diagnostic_evidence/evaluator.py), not the paper's Exact/In-Order/Any-Order Match: evaluate_trajectory(annotation, trajectory) returns milestone_coverage (MC, fraction of milestone groups established), evidence_order_coverage (EOC, fraction established in an admissible causal order), evidence_efficiency (EE, fraction of tool calls that contributed to an established milestone), and process_complete (a boolean gate over a per-case completion_formula, here {"all": ["M1", "M2"]}). The CLI wraps the same function: python -m diagnostic_evidence.cli process-metrics --all --trajectory-root golden-trajectory --annotation-root process-label --app boutique aggregates MC/EOC/EE across every stored golden trajectory for a system. Score outcome (CA/FA/JRA from evaluation.py) and process (MC/EOC/EE from diagnostic_evidence) together, the same principle the paper's Table 4 discussion makes with A@1 versus ZTDR/IAC, just against the metric names the current code actually emits.

7. Maintenance, CI, and version rollback¶

This repository provides none of the usual scaffolding for tracking its own drift, confirmed by inspecting the pinned checkout directly rather than assumed: no .github/workflows, .circleci, .travis.yml, azure-pipelines.yml, or Jenkinsfile exists anywhere in the tree, so there is no upstream CI at all; no tests/ directory or test_*.py file exists outside third-party packages pulled into a virtualenv, so there is no upstream test suite either; and there is no CHANGELOG.md, no git tag, and only a single main branch, so there is no semantic version or release to pin against, only a commit SHA. Practically, that pushes three things onto whoever builds against this benchmark:

There is no CI to depend on, so build your own. The smoke_driver.py and adversarial_driver.py scripts in Section 5 are the only regression check this page can offer; wire them into your own downstream project's CI (clone the pinned SHA, install the three pinned dependencies from Section 1, run both scripts, and assert on the exact fields shown in their transcripts), because nothing upstream will catch a breaking change for you.
Before moving the pin, diff the module tree, not just the README. The pin this page uses, 0faa0b8d5 (2026-07-16), is already one commit behind origin/main's tip. Re-checking this page's claims on 2026-07-17 against a fresh clone found that the very next commit, da213395629c7ce826608408eab0c3bfe88004a2 ("Update evaluation and TrainTicket log data"), renames the entire diagnostic_evidence/ package to cloudops_agent/evaluation_utils/ and updates cloudops_agent/evaluation.py's own imports to match (from diagnostic_evidence.evaluator import evaluate_trajectory becomes from cloudops_agent.evaluation_utils.evaluator import evaluate_trajectory; confirmed by diffing the two commits and by git ls-tree, which shows zero files left under diagnostic_evidence/ at da21339). Anyone who bumps this page's pin forward without re-reading those imports gets ModuleNotFoundError: No module named 'diagnostic_evidence' from every script on this page, including smoke_driver.py, adversarial_driver.py, and the CLI command earlier in this section, which would need to become python -m cloudops_agent.evaluation_utils.cli process-metrics .... Run git diff <old_sha> <new_sha> --stat -M -- cloudops_agent diagnostic_evidence before trusting a new pin, then rerun both driver scripts against it and diff their output against the transcripts on this page.
Rollback means checking out a prior commit, nothing else. With no tags, releases, or package registry entry, "rollback" is git checkout <previous-known-good-SHA> in whatever vendored copy or clone you keep, and there is no pip install cloud-opsbench==X fallback. Keep the known-good SHA and the dependency pins from Section 1 recorded in your own project (a lockfile comment or a line in your own changelog), because the repository keeps neither a changelog nor a version history of its own beyond raw commit messages.

How to extend it with new fault types¶

Fault types enter through Phase 1's <P, A, S> formalization: Prerequisites (required pre-injection cluster state, e.g. a ResourceQuota object), Fault Artifact (the defective YAML or ChaosBlade injection rule), and Activation Sequence (the ordered steps to apply P then A).⁴ Extending the taxonomy means writing a new metadata entry in this shape and passing it through the paper's own triad of gates before it is trustworthy: LLM-as-Judge refinement of the semantic description, human cloud-native expert review of the causal logic, and runtime verification in a sandbox cluster confirming the <P,A,S> tuple reliably triggers the intended symptom.⁴ Skipping the runtime-verification step is the single most likely way a home-grown fault case silently fails to reproduce, since Kubernetes' self-healing behavior (pod rescheduling, automatic restarts) routinely masks naively-specified injections, which is exactly why the paper built a closed Generator/Executor/Verifier loop instead of a one-shot injection script.⁴

To add a new fault category beyond the paper's seven (Admission Control, Scheduling, Startup, Runtime, Service Routing, Performance, Infrastructure), define the symptom-to-verification-step inversion rule that produces T*: the paper's ground-truth synthesis is rule-based, deterministically mapping a known injected fault (e.g., "Added a Taint") to the verification action that reveals it (e.g., "Execute describe node tool"), structured so symptom discovery is a logical prerequisite to root-cause confirmation (Strict Causal Precedence) while tolerating interstitial exploratory actions (Exploratory Noise Tolerance).⁵ The executed code below models exactly this scoring semantics.

How to interpret and report scores honestly¶

Four honesty traps the paper's own data surfaces, worth checking before publishing a Cloud-OpsBench number:

Any-Order beats In-Order for every model in Table 4 (e.g., Qwen3-235B: 0.41 vs 0.38); this is not noise, it quantifies that agents find the right evidence but do not organize it into the linear deductive chain a human expert would.¹¹ Report both, not just whichever is higher.
Low RAR is not automatically good. GPT-4o's near-zero RAR (0.02) correlates with its lowest-tier A@1 (0.49); the paper interprets high RAR in the top performer (DeepSeek-V3.2, 0.11) as functional self-correction, not wasted motion.¹¹ A step-minimization objective would optimize for the wrong thing.
High Tool Relevance does not imply high accuracy. Qwen3-14B's Relevance (0.63) sits between GPT-4o's and GPT-5's, but its A@1 (0.34) is roughly half of GPT-5's (0.67); the bottleneck the paper identifies is information integration (turning a correct tool call's output into a causal chain), a capability Relevance does not measure.¹¹
Stratify by fault explicitness before averaging. Explicit faults (Startup, Runtime, with direct Kubernetes Events like OOMKilled) average A@1 above 0.65; implicit faults (Admission, Performance, requiring cross-layer causal inference) average below 0.36.¹² A single blended A@1 across all 452 cases obscures which failure classes an agent is actually reliable on.

Executed model: trajectory alignment and tool-usage metrics¶

The formulas below (Section 4.2.2) are the paper's own definitions for Exact/In-Order/Any-Order Match and Tool Relevance/Coverage, implemented against a synthetic Admission-Control ground truth modeled on the paper's Case #1 (Fig. 4: a Deployment/ReplicaSet whose Pod creation is rejected at admission time by an exhausted ResourceQuota -- GetResources{deployments} returns "frontend 0/1 ready" and DescribeResource on the ReplicaSet returns "forbidden: exceeded quota"; no Pod object is ever created, which is what makes this an Admission Control fault rather than the Scheduling category's "Pods stay Pending" symptom per Table 2). It is a model of the scoring semantics, not the CrewAI harness or the real 452-case dataset.

"""
Cloud-OpsBench process metrics (paper Section 4.2.2), executed and asserted.
Models: action matching (tool name + critical args must both match), the three
trajectory-alignment strictness levels (Exact / In-Order / Any-Order Match)
scored against a ground truth defined as a *minimal mandatory subsequence*
(Section 3.3.3, "Exploratory Noise Tolerance"), and Tool Relevance / Tool
Coverage computed over the *distinct* tool-call sets S_T the paper defines.
Pure stdlib. This is not the CrewAI evaluation harness; it is a faithful model
of the scoring semantics the paper specifies in prose and formula.
"""
from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Action:
    tool: str
    args: tuple[tuple[str, str], ...]  # critical arguments only, order-independent

    def matches(self, other: "Action") -> bool:
        """Paper 4.2.2: a_i matches g_j iff both tool name AND critical
        arguments are identical -- same tool with a different target does
        not count (models the paper's 'schema hallucination' distinction)."""
        return self.tool == other.tool and self.args == other.args


def exact_match(agent: list[Action], gold: list[Action]) -> bool:
    """Section 4.2.2 (1): replicate the expert's exact sequence, no deviation."""
    if len(agent) != len(gold):
        return False
    return all(a.matches(g) for a, g in zip(agent, gold))


def in_order_match(agent: list[Action], gold: list[Action]) -> bool:
    """Section 4.2.2 (3): gold actions must appear as a subsequence of the
    agent's trajectory in the same relative order; interstitial/extra actions
    are allowed (Section 3.3.3's minimal mandatory subsequence definition)."""
    it = iter(agent)
    return all(any(a.matches(g) for a in it) for g in gold)


def any_order_match(agent: list[Action], gold: list[Action]) -> bool:
    """Section 4.2.2 (2): every gold action present somewhere; order and
    extra actions do not matter."""
    return all(any(a.matches(g) for a in agent) for g in gold)


def distinct_tools(traj: list[Action]) -> set[Action]:
    """S_T: the set of distinct tool calls in trajectory T (Section 4.2.2).
    Repeats of the same call collapse to one element, so redundant retries
    do not inflate Relevance/Coverage."""
    return set(traj)


def relevance(agent: list[Action], gold: list[Action]) -> float:
    """Relevance = |S_agent n S_gold| / |S_agent| (Section 4.2.2)."""
    s_agent, s_gold = distinct_tools(agent), distinct_tools(gold)
    if not s_agent:
        return 0.0
    return len(s_agent & s_gold) / len(s_agent)


def coverage(agent: list[Action], gold: list[Action]) -> float:
    """Coverage = |S_agent n S_gold| / |S_gold| (Section 4.2.2)."""
    s_agent, s_gold = distinct_tools(agent), distinct_tools(gold)
    if not s_gold:
        return 0.0
    return len(s_agent & s_gold) / len(s_gold)


# --- Ground truth for a synthetic Admission-Control fault, modeled on the
# paper's Case #1 (Fig. 4): a Deployment/ReplicaSet whose Pod creation is
# rejected at admission time by an exhausted ResourceQuota (no Pod object
# is ever created -- this is Admission Control, not the Scheduling
# category's "Pods stay Pending" symptom). Minimal mandatory subsequence
# T* per Section 3.3.3: symptom discovery must logically precede
# root-cause confirmation.
GOLD = [
    Action("GetResources", (("resource_type", "pods"),)),
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
    Action("GetResources", (("resource_type", "resourcequota"),)),
]

# 1) A high-fidelity agent trajectory: hits every gold step in order but
# interleaves redundant re-confirmation actions (paper's "redundancy
# paradox", RAR > 0 for the best-scoring model, DeepSeek-V3.2 in Table 4).
agent_thorough = [
    Action("GetResources", (("resource_type", "pods"),)),
    Action("GetResources", (("resource_type", "deployments"),)),          # extra
    Action("GetResources", (("resource_type", "pods"), ("label", "app=frontend"))),  # extra
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
    Action("DescribeResource", (("resource_type", "replicaset"), ("name", "frontend"))),  # extra
    Action("GetResources", (("resource_type", "resourcequota"),)),
]
assert exact_match(agent_thorough, GOLD) is False        # extra steps break exact match
assert in_order_match(agent_thorough, GOLD) is True       # gold subsequence intact, in order
assert any_order_match(agent_thorough, GOLD) is True
assert relevance(agent_thorough, GOLD) == 3 / 6           # 3 of 6 distinct calls are gold
assert coverage(agent_thorough, GOLD) == 3 / 3            # all 3 gold calls hit

# 2) Adversarial: a "lazy leap" agent that jumps straight to the root-cause
# check before establishing the symptom, then backfills. Same *set* of
# actions as the gold, so Any-Order Match still passes, but the causal
# precedence gold requires is violated, so In-Order Match must fail. This
# reproduces the paper's headline finding that Any-Order consistently beats
# In-Order (Table 4, e.g. Qwen3-235B 0.41 vs 0.38, Section 4.4).
agent_leap = [
    Action("GetResources", (("resource_type", "resourcequota"),)),        # jumps to root cause first
    Action("GetResources", (("resource_type", "pods"),)),
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
]
assert in_order_match(agent_leap, GOLD) is False
assert any_order_match(agent_leap, GOLD) is True
assert exact_match(agent_leap, GOLD) is False

# 3) Adversarial: schema hallucination / wrong target. Same tool name as the
# gold's root-cause check but a different critical argument (wrong
# namespace), so it must NOT count as a match even though the tool string is
# identical -- this is what the paper's Invalid Action Count / schema
# hallucination failure mode (Section 4.5, Fig. 4 Case #3) actually breaks.
agent_wrong_target = [
    Action("GetResources", (("resource_type", "pods"),)),
    Action("DescribeResource", (("resource_type", "deployments"), ("name", "frontend"))),
    Action("GetResources", (("resource_type", "resourcequota"), ("namespace", "staging"))),  # wrong ns
]
assert any_order_match(agent_wrong_target, GOLD) is False
assert coverage(agent_wrong_target, GOLD) == 2 / 3
assert relevance(agent_wrong_target, GOLD) == 2 / 3

# 4) Adversarial edge case: Zero-Tool Diagnosis (ZTDR, Section 4.2.2). The
# agent emits a final answer with no tool calls at all -- must not crash on
# an empty trajectory and must score as a total miss on every metric.
agent_ztdr: list[Action] = []
assert exact_match(agent_ztdr, GOLD) is False
assert in_order_match(agent_ztdr, GOLD) is False
assert any_order_match(agent_ztdr, GOLD) is False
assert relevance(agent_ztdr, GOLD) == 0.0
assert coverage(agent_ztdr, GOLD) == 0.0

# 5) Redundancy must not inflate Relevance/Coverage: repeating the same
# correct call 5 times collapses to one element of S_agent (Section 4.2.2
# defines S_T over distinct calls), so Relevance is unchanged versus a
# single clean hit, not diluted or amplified by retry count.
agent_repetitive = GOLD + [GOLD[-1]] * 4
assert distinct_tools(agent_repetitive) == distinct_tools(GOLD)
assert relevance(agent_repetitive, GOLD) == 1.0
assert coverage(agent_repetitive, GOLD) == 1.0
assert exact_match(agent_repetitive, GOLD) is False  # length differs (5 extra steps)

print("thorough  : exact=%s in_order=%s any_order=%s rel=%.2f cov=%.2f" % (
    exact_match(agent_thorough, GOLD), in_order_match(agent_thorough, GOLD),
    any_order_match(agent_thorough, GOLD), relevance(agent_thorough, GOLD), coverage(agent_thorough, GOLD)))
print("leap      : in_order=%s any_order=%s (Any-Order > In-Order reproduced)" % (
    in_order_match(agent_leap, GOLD), any_order_match(agent_leap, GOLD)))
print("wrong_tgt : any_order=%s rel=%.2f cov=%.2f (schema-mismatch action excluded)" % (
    any_order_match(agent_wrong_target, GOLD), relevance(agent_wrong_target, GOLD), coverage(agent_wrong_target, GOLD)))
print("ztdr      : exact=%s in_order=%s any_order=%s rel=%.2f cov=%.2f" % (
    exact_match(agent_ztdr, GOLD), in_order_match(agent_ztdr, GOLD), any_order_match(agent_ztdr, GOLD),
    relevance(agent_ztdr, GOLD), coverage(agent_ztdr, GOLD)))
print("repetitive: distinct_calls=%d rel=%.2f cov=%.2f (retries do not inflate score)" % (
    len(distinct_tools(agent_repetitive)), relevance(agent_repetitive, GOLD), coverage(agent_repetitive, GOLD)))
print("all trajectory-metric assertions passed")

Executed output:

thorough  : exact=False in_order=True any_order=True rel=0.50 cov=1.00
leap      : in_order=False any_order=True (Any-Order > In-Order reproduced)
wrong_tgt : any_order=False rel=0.67 cov=0.67 (schema-mismatch action excluded)
ztdr      : exact=False in_order=False any_order=False rel=0.00 cov=0.00
repetitive: distinct_calls=3 rel=1.00 cov=1.00 (retries do not inflate score)
all trajectory-metric assertions passed

Failure modes¶

Pitfall	Cause	Fix
Reporting a single A@1 as "the" score	Averages across explicit faults (A@1 > 0.65, e.g. OOMKilled) and implicit faults (A@1 < 0.36, e.g. Admission/Performance) mask a bimodal distribution.¹²	Break results out by fault category before comparing agents or claiming production readiness.
Optimizing an agent to minimize Steps or RAR	The paper's own data shows the highest-accuracy model (DeepSeek-V3.2) has the most steps (10.0) and highest RAR (0.11); GPT-4o's near-zero RAR (0.02) correlates with its weakest A@1 (0.49).¹¹	Treat redundant verification as a feature to preserve, not a cost to eliminate, unless MTTI is a hard product constraint.
Trusting Tool Relevance as a proxy for diagnostic quality	Qwen3-14B matches GPT-4o's Relevance (0.63) but trails badly on A@1 (0.34 vs 0.49); tool selection and information integration are separate capability layers.¹¹	Score outcome and process metrics together; never gate on Relevance/Coverage alone.
Assuming State Snapshot faithfully models live-cluster stochasticity	The paradigm freezes the crime scene by design (Threats to Validity, Construct Validity); it explicitly cannot capture real-time state transitions or trial-and-error remediation.¹³	Pair with a live-environment benchmark like AIOpsLab before shipping an agent that must also act on a cluster, not just diagnose it.
Skipping runtime verification when authoring new fault cases	Kubernetes' self-healing (pod rescheduling, automatic restarts) silently masks naive injections; the paper's own Generator/Executor/Verifier loop exists because of this.⁴	Run the sandbox-cluster verification step (the paper's third gate) before committing a new `<P,A,S>` tuple to a fault knowledge base.
Citing the 40-fault-type taxonomy as covering "cloud failures" generally	Every fault type and tool is Kubernetes-specific (Table 2, Table 3); the authors flag monolithic/serverless transfer as unverified.¹³	Scope claims to Kubernetes-native RCA; validate separately before extending to other platforms.
Assuming the repository still runs on CrewAI or scores Exact/In-Order/Any-Order Match	As documented above, the current `main` branch is a custom single-agent ReAct runtime scored by milestone coverage (MC/EOC/EE), not the paper's harness or metrics; the two have diverged since publication.	Read `cloudops_agent/runtime/` and `diagnostic_evidence/evaluator.py` directly before building against this benchmark; do not assume paper section numbers describe current code.
Loading `process-label/*/milestone.json` straight into `diagnostic_evidence.schema.CaseAnnotation.from_dict`	The shipped milestone files omit the `description` field `Milestone.from_dict` requires, so calling `CaseAnnotation.from_dict` directly on the raw file raises `KeyError: 'description'` on every case as of `0faa0b8d5`. This is a caller error, not a repository defect: confirmed by reading `cloudops_agent/evaluation.py`, no gap to file upstream.	Use the repository's own `cloudops_agent.evaluation.load_process_annotation()` instead of calling `CaseAnnotation.from_dict` directly; it already defaults the missing `description`/`role` fields before constructing the annotation, exactly as shown in "How to run it" above.
Bumping the pinned commit without re-checking module paths	The repository has no tags, releases, or CI; the commit immediately after this page's pin (`da21339`, 2026-07-17) renamed the whole `diagnostic_evidence/` package to `cloudops_agent/evaluation_utils/` and updated `cloudops_agent/evaluation.py`'s own imports to match, confirmed by diffing the two commits directly. Every import in this page's driver scripts breaks silently on a naive pin bump.	Diff `cloudops_agent` and `diagnostic_evidence` between old and new SHA before moving the pin, then rerun `smoke_driver.py`/`adversarial_driver.py` against the new commit and compare output to this page's transcripts; see "Maintenance, CI, and version rollback" above.

References¶

Wang, Yu, Huang, Wang, Huang, Chen, Lyu, Cloud-OpsBench: A Reproducible Benchmark for Agentic Root Cause Analysis in Cloud Systems (arXiv 2603.00468): https://arxiv.org/abs/2603.00468
PDF: https://arxiv.org/pdf/2603.00468
Code and dataset, pinned at commit 0faa0b8d53d16b153b7300c03f5a29791aea9cb9 (2026-07-16; 754 cases, 57 fault types, two systems as of this commit, versus the paper's 452 cases/40 types/one system): https://github.com/LLM4Ops/Cloud-OpsBench
AIOpsLab (MLSys 2025), the live-environment benchmark Cloud-OpsBench positions itself against: https://arxiv.org/abs/2501.06706
OpenRCA (ICLR 2025), the static-artifact benchmark with generative (code-synthesis) tool use: https://openreview.net/forum?id=M4qNIzQYpd
Google Online Boutique microservices demo (the workload used as the testbed): https://github.com/GoogleCloudPlatform/microservices-demo
ChaosBlade fault-injection tool: https://chaosblade.io/

Section 3.1, "Agentic RCA Task Formulation": ground truth tuple R* = <S, C, R> (Stage, Component, Root Cause); task modeled as f: <A_alert, E_snapshot> -> <T, R_hat> (Eq. 1); trajectory T = [(t_1,a_1,o_1), ..., (t_n,a_n,o_n)] of thought/action/observation steps. ↩
Section 3.3.3, "2. State Snapshot": exhaustive parameter sweep against tools T1-T10 (Table 3); "each fault case generates 487 distinct tool invocations" on average; valid queries hit deterministic entries, invalid queries map to realistic Not Found errors; reasoning search space stated as 487^N with N approx 10 required interdependent steps, argued to make random guessing statistically impossible. ↩↩
Section 3.3.1, "Benchmark Testbed": Huawei Cloud ECS, Kubernetes v1.31 on 4 instances, Prometheus + Istio observability, Google Online Boutique (11 microservices, gRPC) driven by Locust, ChaosBlade for infrastructure-level perturbation; Table 2 taxonomy: 7 fault categories, 40 fault types, 452 total cases (Admission Control 58, Scheduling 164, Startup 62, Runtime 45, Service Routing 54, Performance 21, Infrastructure 48). ↩↩
Section 3.3.2 (Phase 1) and 3.3.3 (Phase 2): fault metadata as <Semantic Metadata, <P,A,S>> (Prerequisites, Fault Artifact, Activation Sequence); validation triad LLM-as-Judge (Gemini 3 Pro) -> human expert -> sandbox runtime verification; Generator/Executor/Verifier closed loop, Verifier re-triggers Generator on injection masking (e.g. taint bypassed by rescheduling). ↩↩↩↩↩↩
Section 3.3.3, "3. Ground Truth Annotation": G* = <R*, T*>; T synthesized by rule-based inversion of the injection logic; two design principles named verbatim: Strict Causal Precedence (Anti-Guessing) and Exploratory Noise Tolerance (Pro-Redundancy), the latter defining T as "a minimal mandatory subsequence rather than a rigid script." ↩
Section 5.1, "Significance and Impact": "Democratizing Research via Zero-Cost Replay" paragraph states prior live-cluster benchmarks require "often hours per run" while Cloud-OpsBench "enables a full-scale benchmark run in seconds on a standard laptop"; Section 5.2 frames the benchmark as a data engine for SFT bootstrapping via harvested trajectories. ↩↩
Section 2.3, "Gap 2: The Reproducibility Challenge in Dynamic Environments": network-jitter example (a 500ms delay injection triggering cascading timeouts in one run, seamless retry in another) used to argue live A/B testing across models is invalid. ↩
Section 4.2, "Evaluation Metrics" and 4.2.2 "Process-based Metrics": A@k formula (Section 4.2.1); trajectory alignment three strictness levels (Exact/Any-Order/In-Order Match) with formal match condition (tool name + critical arguments identical); Relevance and Coverage formulas over distinct tool-call sets S_T. ↩↩
Table 4, "Performance comparison of LLMs in Agentic RCA on outcome and process metrics": DeepSeek-V3.2 A@1 0.73, A@3 0.79, Steps 10.0, Coverage 0.88, RAR 0.11; GPT-4o A@1 0.49, MTTI 23.27, Steps 5.67, RAR 0.02; GPT-5 A@1 0.67, Relevance 0.65, Steps 5.57; Qwen3-14B Relevance 0.63, IAC 0.40, A@1 0.34; Qwen3-8B A@1 0.21; Claude-4-Sonnet A@1 0.5, ZTDR 0.32, Steps 4.25; discussed in Sections 4.3-4.5. ↩↩↩
Table 5, "Performance comparison of knowledge-enhanced agents" and Section 4.6 "RQ4: Knowledge Efficacy": GPT-4o Base A@1 0.49 -> ICL 0.7 vs RAG 0.61; Qwen3-14B Base A@1 0.34 -> ICL 0.71 (IAC 0.40 -> 0.29) vs RAG 0.5; Qwen3-235B CoT degrades A@1 from 0.50 to 0.47. ↩
Section 4.4, "RQ2: RCA Process Alignment": Qwen3-235B Any-Order 0.41 vs In-Order 0.38; DeepSeek-V3.2 RAR 0.11 termed a "redundancy paradox" and functional self-correction; GPT-4o RAR 0.02 with lower accuracy; DeepSeek-V3.2 called "Coverage-Centric," GPT-5 called "Relevance-Centric"; Qwen3-14B Relevance 0.63 vs A@1 0.34 gap attributed to "information integration," not tool selection. ↩↩↩↩↩
Section 4.3, "RQ1: RCA Outcome Effectiveness," "Performance stratification is determined by symptom explicitness and observability": explicit faults (Startup, Runtime) "Avg A@1 > 0.65"; implicit faults (Admission, Performance) "Avg A@1 < 0.36." ↩↩
Section 5.3, "Threats to Validity": External Validity notes Kubernetes-specific toolsets may not generalize to monolithic/serverless platforms; Construct Validity states the State Snapshot paradigm "restricts the agent's ability to observe real-time state transitions" and explicitly scopes the benchmark to reasoning over existing evidence, "distinguishing itself from downstream tasks that require trial-and-error remediation or active environmental intervention." ↩↩↩↩
Section 4.1, "Experimental Setup": Python 3.10, CrewAI orchestration, Pydantic tool schemas, Langfuse observability; seven models in two tiers (LLM tier: GPT-5, GPT-4o, Claude-4-Sonnet, DeepSeek-V3.2, Qwen3-235B; SLM tier under 20B: Qwen3-14B, Qwen3-8B). ↩
Table 3, "Description of specialized diagnostic tools in Cloud-OpsBench": T1 GetResources, T2 DescribeResource, T3 GetAppYAML, T4 GetServiceDependencies, T5 GetRecentLogs, T6 CheckServiceConnectivity, T7 GetClusterConfiguration, T8 GetAlerts, T9 GetErrorLogs, T10 CheckNodeServiceStatus. ↩