Skip to content
Markdown

STRATUS: transactional non-regression for autonomous SRE

Scope: STRATUS (arXiv 2506.02009, NeurIPS 2025), a multi-agent LLM system that mitigates live cloud failures without a human in the loop, built around a formalized safety property, Transactional Non-Regression (TNR), that bounds how much worse an autonomous mitigation attempt can make the system before an undo restores the prior state. Covers the environment and severity model, the TNR assumptions and transaction semantics with proof sketch, the CrewAI-based state-machine implementation (confinement, stack-based undo, oracles), and the AIOpsLab/ITBench results including the undo-and-retry ablation. The benchmark suites it runs against are AIOpsLab and, by name only in this page (not separately covered in this KB), ITBench; the pipeline-shaped sibling in this KB is OpsAgent; the field view of guardrails and telemetry compaction for agentic ops is agentic AIOps; the general oversight-ladder pattern STRATUS's "hybrid operations" hook could plug into is risk-tiered approval gates.

Independently verified here: the full paper text (PDF via arXiv, extracted with pdftotext -layout), all tables and figures cited below, and that the code repository https://github.com/xlab-uiuc/stratus is public (confirmed via the GitHub API, "private": false, 2026-07-16; contents beyond existence were not audited). The Python block is executed and asserted; it models the TNR transaction mechanism (checkpoint/execute/commit-abort, the bounded risk window, the undo-and-retry loop) and reproduces Table 1's four worked examples numerically. It does not run STRATUS, CrewAI, or any LLM. Note a wording quirk in the paper itself: the abstract and introduction (Sec 1) say "Transactional No-Regression"; Section 3 and the contributions list say "Transactional Non-Regression"; both refer to the same TNR property, and this page uses "Non-Regression" since that is what the formal treatment (Sec 3) uses throughout.

What it is

STRATUS is an LLM-based multi-agent system that performs the full SRE lifecycle, detection, localization, root-cause analysis (RCA), and mitigation, against live emulated cloud services, with the explicit goal of acting autonomously rather than assisting a human operator.1 It models a target cloud as an environment E with a state set S, a crash state ⊥, and a severity metric μ(s) = w1|A| + w2|V| + w3|L| over alert count, SLA violations, and unhealthy-node capacity loss, with μ(⊥) = ∞.2 Four agents divide the work: a Detection agent αD establishes the initial error state; a Diagnosis agent αG handles both localization and RCA; a Mitigation agent αM plans and executes state-changing commands; and an Undo agent αU reverts a mitigation attempt if it fails. Detection and diagnosis are read-only (Aread); only mitigation and undo can mutate the system (Awrite, Aundo), issued through an Agent-Computer Interface (ACI) built on kubectl.3

The paper's central contribution is not the four-agent split itself but a safety specification for it: Transactional Non-Regression (TNR). TNR wraps every mitigation attempt in transaction semantics, checkpoint the entry state, execute a bounded sequence of commands, then commit only if the system did not crash and its severity did not increase versus the checkpoint, otherwise abort and undo, restoring the checkpoint exactly. The paper proves (Lemma 3.1, by induction) that under this discipline every externally visible state satisfies μ(s) ≤ b, the severity at the moment the failure was first detected, no matter how bad the system gets inside an aborted transaction's hidden execution path.4 This is what licenses STRATUS's headline mechanism: safe retry. Because a failed mitigation attempt is invisible to the rest of the system, STRATUS can try an ambitious plan, watch it fail, undo it for free, and try something else, up to nine times per problem.12

Why use it

  • The safety property is the deliverable, not the agent count. TNR is what makes "let an LLM run mutating kubectl commands against production with no human approval" a defensible sentence: the worst case is bounded by the state the system was already in, formally, not just empirically.4
  • Undo-and-retry is measurably the source of STRATUS's advantage, not multi-agent framing alone. Ablating retry entirely on AIOpsLab (13 mitigation problems, GPT-4o) drops success from 69.2% to 15.4%; keeping retry but removing undo (retrying from wherever the failed attempt left the system, instead of rolling back) only reaches 23.1%, worse time (1221.5s vs 811.9s) and worse cost ($0.929 vs $0.877) for a worse outcome. The paper's reading: failed mitigations that are not undone make the system harder to save on the next attempt.12
  • It beats the field's reference agents by at least 1.5x on mitigation, the task that actually requires safety. STRATUS (GPT-4o) solves 69.2% of AIOpsLab's 13 mitigation problems versus 46.2% for the next-best baseline (AOL-agent), and 50.0% of ITBench's 18 versus 9.2% (ITB-agent); the smallest ratio across all model/benchmark pairs in Table 2 is 1.5x (both AIOpsLab GPT-4o and AIOpsLab Llama 3.3), consistent with the abstract's "at least 1.5 times" claim.11
  • The confinement layer prevents concrete hangs, not just abstract risk. Interactive and stdin-blocking kubectl invocations (exec -it, apply -f -) would have hung the agent session; the paper measured that 10.5% of the commands STRATUS (GPT-4o) issued, and 9.7% of STRATUS (Llama 3.3)'s, would have caused exactly this failure mode absent the confinement rules.8
  • The cost is honestly reported, and it is real. Safety buys retries, and retries cost tokens and wall-clock time: STRATUS (GPT-4o) takes 3.6x the time (811.9s vs 223.3s) and 4.3x the cost ($0.877 vs $0.206) of AOL-agent on the same AIOpsLab mitigation set. The paper frames this as still cheap relative to human SRE time, without benchmarking that comparison directly.11

When to use it (and when not)

  • Use it as the reference design for autonomous mitigation against declarative, reconciliation-based platforms (Kubernetes, and by the paper's own analogy Borg/Twine/ECS-style systems), where "faithful undo" is tractable because the platform already tracks and can restore prior object state.7
  • Use it where the cost of an autonomous action making things measurably worse is the actual blocker to deployment, and a formally bounded worst case (never below the pre-incident baseline, in the externally visible trace) is worth paying 3-4x the latency and cost of a baseline agent that, per the paper, lacks TNR's safe-retry capability.
  • Do not treat ITBench's headline mitigation number as proof of robust fixes. In 8 of ITBench's 18 mitigation problems, STRATUS wins because the benchmark's fault injector cannot re-target a pod after it restarts, so repeatedly restarting failing pods clears the alert without addressing the injected fault; the paper states plainly this strategy would not work against persistent faults like misconfigurations or hardware defects.12
  • Do not deploy it expecting concurrent multi-writer safety out of the box. TNR's Writer Exclusivity (A1) is a single readers-writer lock; the current deployment model is one STRATUS instance per cloud system, one namespace at a time. The paper's resource-aware Coordination Controller for concurrent writers is a described extension, not an implemented one.6
  • Do not trust the RCA success numbers as a diagnosis-quality signal without inspection. AIOpsLab's RCA oracle uses non-mutually-exclusive category labels; the paper documents a case where STRATUS's answer ("Application" layer, "Network/Storage Issue" fault type) is arguably more precise than the oracle's expected answer ("Virtualization", "Misconfiguration") but is scored wrong regardless.13
  • Do not assume the rule-based confinement layer catches every destructive action; the authors state it directly as a limitation, not a completeness guarantee.7

Architecture

flowchart TB
  FAIL["Failure in cloud E<br/>baseline severity b = mu(se0)"] --> DET["Detection agent aD (Aread only)<br/>establishes initial error state se0"]
  DET --> DIAG["Diagnosis agent aG (Aread only)<br/>localization + RCA, trace-based bootstrapping"]
  DIAG --> CKPT["Mitigation agent aM acquires A-Lock<br/>R1: checkpoint spre"]
  CKPT --> EXEC["R2: execute a1..ak, k less-or-equal K=20<br/>via NL2Kubectl + LintingTool + kubectl --dry-run"]
  EXEC --> CLOUD["Cloud E (kubectl ACI, confinement-filtered)"]
  CLOUD --> ORACLE{"R3: oracle check on spost<br/>alert clearance / user requests / system health"}
  ORACLE -->|"commit: spost != crash AND mu(spost) less-or-equal mu(spre)"| VIS["externally visible state advances to spost"]
  ORACLE -->|"abort"| UNDO["Undo agent aU acquires A-Lock<br/>Faithful Undo: U(spost) = spre via action stack"]
  UNDO --> VISU["externally visible state restored to spre"]
  VIS -->|"oracles fail: still unhealthy"| CKPT
  VISU -->|"retry, up to 9x per problem"| CKPT
  VIS -->|"all oracles pass"| DONE["NotificationTool: mitigation succeeded"]

Writer Exclusivity (A1) is enforced two ways: role-based sandboxing (detection and diagnosis agents cannot issue mutating commands at all) and state-machine scheduling that serializes the writer agents (αM, αU), so at most one is active at a time, a readers-writer lock over the whole system.7 Faithful Undo (A2) is realized as a stack-based rollback: every mutating command pushes a (command, resource_type, prior_state) tuple, and undo mechanically pops the stack in LIFO order, restoring each affected Kubernetes object; irrecoverable actions (an outright file delete) are either rejected or rewritten into a recoverable equivalent (move to a backup volume instead).7 Bounded Risk Window (A3) caps a single transaction at K=20 commands, set from an empirical sweep (3, 5, 10, 15, 20, 30) where STRATUS (GPT-4o) jumps from 0% to 53.53% success moving from K=3 to K=10 and plateaus past 15.14

How to use it

STRATUS is not a library you import into an existing pipeline; it is a reference agent implementation over CrewAI, evaluated against two benchmark harnesses (AIOpsLab, ITBench) that already provide the live Kubernetes environments, fault injectors, and oracles.10 The practical entry points from the public repository are the same shape as any AIOpsLab agent (see AIOpsLab's get_action contract): point STRATUS's Detection agent at your telemetry backend, wire the Mitigation agent's NL2Kubectl tool to your cluster's kubectl context, and set the confinement rules (Table 5 in the paper: block namespace deletion, interactive flags, stdin redirection, shell pipes and compound commands, command substitution, and flow control, for every command STRATUS's linter sees) before granting write access to anything real.8 The paper's own evaluation runtime is a useful sizing reference: each of the two benchmark suites took STRATUS 15-20 machine hours to finish end to end.10

Two implementation choices matter operationally. First, STRATUS disables direct raw-metric consumption by the agents; numerical telemetry is pre-processed into filtered, aggregated, human-readable summaries before it reaches the model, the same telemetry-compaction discipline covered in agentic AIOps.9 Second, termination is a combined-oracle decision, not a single check: alert clearance, absence of 4xx/5xx user-facing errors, and Kubernetes-level system health (pods, volumes) are each "weak oracles" individually, and STRATUS only concludes success when all three pass, illustrated in the paper by a case where the cluster looked healthy at the orchestration level while the workload oracle still measured 115 failed responses out of 117.9

How to develop with it

The transaction mechanism, checkpoint, execute, commit-or-abort, undo, is small enough to validate directly without CrewAI or an LLM in the loop, and internalizing it is the prerequisite for extending STRATUS safely (a new writer agent, a different severity metric, a different confinement policy). The executed model below implements run_transaction per the paper's R1-R3 rules and visible_mu per A2's Faithful Undo, reproduces all four of Table 1's worked mitigation plans numerically5, exercises two adversarial cases (a transaction that exceeds the bounded risk window K, and a mid-transaction crash that must abort regardless of what a later action would have done), checks Lemma 3.1's invariant over 200 randomized transactions, and contrasts TNR's undo-and-retry against the paper's "naive retry w/o undo" ablation on a hand-built compounding-failure scenario:

# stratus_tnr_model.py - validated: implements the paper's Transactional
# Non-Regression (TNR) transaction semantics (Sec 3.1.1-3.1.3: A1-A3, R1-R3,
# Table 1, Lemma 3.1) and models the qualitative ablation finding that
# undo-less retry compounds severity (Sec 5, "Naive retry w/o undo"). Pure
# stdlib; models the mechanism, does not run STRATUS or CrewAI.
from __future__ import annotations

import math
import random
from typing import Sequence

CRASH = math.inf  # mu(perp) = infinity, paper Sec 2


def run_transaction(mu_pre: float, deltas: Sequence[float], K: int) -> tuple[list[float], float, bool]:
    """R1 checkpoint (mu_pre) -> R2 execute a1..ak, deltas applied sequentially
    -> R3 commit iff not crashed and mu_post <= mu_pre (Sec 3.1.2-3.1.3).
    A3 bounds the transaction length to K (bounded risk window)."""
    assert 1 <= len(deltas) <= K, f"A3 violated: transaction length {len(deltas)} exceeds K={K}"
    hidden = [mu_pre]
    for d in deltas:
        hidden.append(hidden[-1] + d)
    mu_end = hidden[-1]
    committed = (mu_end < CRASH) and (mu_end <= mu_pre)
    return hidden, mu_end, committed


def visible_mu(mu_pre: float, mu_end: float, committed: bool) -> float:
    """R3 + A2 (Faithful Undo): commit exposes mu_end; abort restores mu_pre
    exactly, so the hidden path is invisible to other agents (Sec 3.1.2-3.1.3)."""
    return mu_end if committed else mu_pre


def retry_loop(mu_baseline: float, attempts: Sequence[Sequence[float]], K: int,
                use_undo: bool) -> list[float]:
    """TNR undo-and-retry (use_undo=True) re-checkpoints every attempt from the
    last externally visible state; the ablation's naive retry w/o undo
    (use_undo=False) chains from wherever the failed attempt's hidden path
    ended, per Sec 5's ablation description."""
    trace = [mu_baseline]
    mu_pre = mu_baseline
    for deltas in attempts:
        _, mu_end, committed = run_transaction(mu_pre, deltas, K)
        mu_pre = visible_mu(mu_pre, mu_end, committed) if use_undo else mu_end
        trace.append(mu_pre)
    return trace


# --- 1) Table 1 replication: the paper's four worked mitigation plans -------
rebalance = run_transaction(12.0, [18 - 12, 9 - 18], K=20)
assert rebalance[:2] == ([12.0, 18.0, 9.0], 9.0) and rebalance[2] is True
assert visible_mu(12.0, rebalance[1], rebalance[2]) == 9.0  # visible: 12 -> 9

upgrade = run_transaction(15.0, [22 - 15, 11 - 22], K=20)
assert upgrade[:2] == ([15.0, 22.0, 11.0], 11.0) and upgrade[2] is True
assert visible_mu(15.0, upgrade[1], upgrade[2]) == 11.0  # visible: 15 -> 11

bad_image = run_transaction(15.0, [24 - 15, 30 - 24], K=20)
assert bad_image[:2] == ([15.0, 24.0, 30.0], 30.0) and bad_image[2] is False
assert visible_mu(15.0, bad_image[1], bad_image[2]) == 15.0  # aborts: visible 15 -> 15

hotfix_ok = run_transaction(15.0, [10 - 15], K=1)      # x=10 <= 15: commits
assert hotfix_ok[2] is True and visible_mu(15.0, hotfix_ok[1], True) == 10.0
hotfix_bad = run_transaction(15.0, [20 - 15], K=1)     # x=20 > 15: aborts
assert hotfix_bad[2] is False and visible_mu(15.0, hotfix_bad[1], False) == 15.0

# --- 2) Adversarial: A3's bounded risk window rejects an oversized transaction,
#    and a mid-transaction crash (mu -> CRASH) always aborts regardless of a
#    later delta that would otherwise look like a recovery. --------------------
try:
    run_transaction(15.0, [1, 1, 1], K=2)
    raise SystemExit("expected AssertionError for A3 violation")
except AssertionError as e:
    assert "A3 violated" in str(e)

crash_then_recover = run_transaction(15.0, [CRASH, -1000.0], K=20)
assert crash_then_recover[1] == CRASH and crash_then_recover[2] is False
assert visible_mu(15.0, crash_then_recover[1], crash_then_recover[2]) == 15.0

# --- 3) Lemma 3.1: over a long trajectory of interleaved committing and
#    aborting transactions (including occasional injected crashes), the
#    externally visible severity never exceeds the baseline b, regardless of
#    how bad the hidden path gets inside any single transaction. -------------
rng = random.Random(7)
b = 20.0
mu_visible = b
lemma_trace = [mu_visible]
for _ in range(200):
    k = rng.randint(1, 5)
    deltas = [rng.uniform(-15, 25) for _ in range(k)]
    if rng.random() < 0.05:
        deltas[rng.randrange(k)] = CRASH  # occasional injected crash attempt
    _, mu_end, committed = run_transaction(mu_visible, deltas, K=20)
    mu_visible = visible_mu(mu_visible, mu_end, committed)
    lemma_trace.append(mu_visible)
assert all(m <= b for m in lemma_trace), "TNR invariant violated: visible severity exceeded baseline"

# --- 4) TNR undo-and-retry vs the ablation's "naive retry w/o undo" (Sec 5):
#    three hallucinated fixes followed by one real fix. TNR always re-checks
#    out from the last visible (undone) state, so severity never exceeds
#    baseline; naive retry chains off each failed attempt's hidden end state,
#    so severity compounds before the real fix ever runs. ---------------------
bad_attempts = [[8.0, 6.0], [7.0, 5.0], [9.0, 4.0], [-40.0]]
tnr_trace = retry_loop(b, bad_attempts, K=20, use_undo=True)
naive_trace = retry_loop(b, bad_attempts, K=20, use_undo=False)
assert max(tnr_trace) <= b, "TNR retry must never exceed baseline"
assert max(naive_trace) > b, "naive retry should compound past baseline in this adversarial case"
assert naive_trace[-1] > tnr_trace[-1], "naive retry ends worse off than TNR retry"

print("Table 1 replication (hidden mu-path -> commit? -> visible):")
print(f"  node drain/rebalance   {rebalance[0]} commit={rebalance[2]} visible=12->9")
print(f"  rolling upgrade        {upgrade[0]} commit={upgrade[2]} visible=15->11")
print(f"  bad image attempted    {bad_image[0]} commit={bad_image[2]} visible=15->15")
print(f"  hotfix K=1 (x=10)      {hotfix_ok[0]} commit={hotfix_ok[2]}")
print(f"  hotfix K=1 (x=20)      {hotfix_bad[0]} commit={hotfix_bad[2]}")
print("A3 bounded-risk-window violation raised AssertionError as expected")
print(f"mid-transaction crash always aborts: visible stays {visible_mu(15.0, crash_then_recover[1], crash_then_recover[2])}")
print(f"Lemma 3.1 over 200 transactions: max visible mu = {max(lemma_trace):.2f} <= baseline b = {b}")
print(f"retry comparison: TNR trace   = {[round(x, 1) for x in tnr_trace]}")
print(f"retry comparison: naive trace = {[round(x, 1) for x in naive_trace]}")
print("all assertions passed")

Executed output:

Table 1 replication (hidden mu-path -> commit? -> visible):
  node drain/rebalance   [12.0, 18.0, 9.0] commit=True visible=12->9
  rolling upgrade        [15.0, 22.0, 11.0] commit=True visible=15->11
  bad image attempted    [15.0, 24.0, 30.0] commit=False visible=15->15
  hotfix K=1 (x=10)      [15.0, 10.0] commit=True
  hotfix K=1 (x=20)      [15.0, 20.0] commit=False
A3 bounded-risk-window violation raised AssertionError as expected
mid-transaction crash always aborts: visible stays 15.0
Lemma 3.1 over 200 transactions: max visible mu = 20.00 <= baseline b = 20.0
retry comparison: TNR trace   = [20.0, 20.0, 20.0, 20.0, -20.0]
retry comparison: naive trace = [20.0, 34.0, 46.0, 59.0, 19.0]
all assertions passed

The retry comparison is the point worth sitting with: after three bad attempts, TNR's visible trace never leaves the baseline (20.0 throughout), because each abort is invisible; the naive trace climbs to 59.0, a system nearly three times as sick as when detection started, before the fourth, genuinely good, attempt only claws it back to 19.0, still worse than TNR's -20.0 (fully resolved) from the same four attempts. This is the executed shape of the paper's own finding on real problems: undo-less retry from AIOpsLab's ablation reaches only 23.1% success against TNR's 69.2%.12

To extend STRATUS: a new writer agent must acquire the A-Lock like αM/αU (no exceptions, or A1 is void); a new severity metric only needs to preserve μ(⊥) = ∞ and monotonicity under the intended alert/violation/capacity signals for Lemma 3.1's proof to still apply; and a new confinement rule set should be additive to Table 5, not a replacement, since the paper's own limitation is incompleteness, not overreach.7

Running it in production

The paper does not describe a customer-facing production deployment; the evaluation is against two benchmark harnesses' emulated clouds (DeathStarBench SocialNetwork/HotelReservation via AIOpsLab, and ITBench's scenario set), 15-20 machine hours per suite.10 What it does specify, and what a real rollout should copy, is the guardrail stack that makes autonomous write access defensible: role-based confinement (readers cannot mutate, ever), command-level confinement (a static linter rejects pipes, compound commands, command substitution, and interactive flags before anything reaches the cluster), kubectl --dry-run previews where the API server supports them, and the combined-oracle termination check (alert clearance, user-facing error rate, and object-level health, all three, not any one).789 STRATUS's own design explicitly supports a lower-autonomy fallback: "hybrid operations where writer agents can request human approval before execution, with interceptable action stacks and tool interfaces", the natural hook for wiring a risk-tiered approval gate in front of αM for actions your policy classifies above auto/validate.7

Sizing and cost: at GPT-4o, budget roughly 800-1,700 seconds and $0.88-$6.11 per mitigation problem depending on benchmark complexity, with retries (up to 9) accounting for most of both. Whether GPT-4o-mini saves money is benchmark-dependent, not a general property of the cheaper model: on AIOpsLab it cuts cost by roughly 24x ($0.036 vs $0.877) because its step count only grows about 2.7x (125.7 vs 46.3 steps) to compensate for weaker per-step performance; on ITBench it costs 53% more ($9.38 vs $6.11) because its step count blows up about 4x (468.9 vs 115.7 steps), outrunning the per-token price cut. Success rate falls by more than half on both benchmarks (69.2% to 23.1% on AIOpsLab, 50.0% to 19.4% on ITBench). So model choice is a genuine success/cost trade, and on the harder benchmark it is not even a cost saving.11 The current concurrency model is one STRATUS instance per managed system (one namespace, single writer at a time); running many instances against overlapping infrastructure needs the Sec 3.3 resource-locking extension the paper describes but does not implement, since TNR's guarantee as published does not extend to concurrent writers without it.6

Failure modes

Pitfall Cause Fix
Undo is assumed faithful but is not always achievable Complex or externally-visible side effects (application state, third-party API calls) do not reduce to a simple state-object rollback; the paper names this a practical limitation, not a solved problem Scope autonomous write access to reconciliation-based resources (Kubernetes objects and similar) where rollback is tractable; require human approval for anything with side effects outside the platform's own state model
A benchmark "win" can be a benchmark artifact In 8 of 18 ITBench mitigation problems, STRATUS succeeds by restarting failing pods, which clears the alert because the fault injector cannot re-target a restarted pod, not because the underlying fault (a persistent misconfiguration or bad image) was fixed Do not report ITBench mitigation success as evidence of fixing persistent faults; re-verify against faults that are known to survive a pod restart before trusting the number
Weaker models blow the retry budget without matching gains STRATUS (GPT-4o-mini) took 3,557.9s and 125.7 steps on AIOpsLab mitigation for 23.1% success, versus GPT-4o's 811.9s/46.3 steps for 69.2%; more retries did not close the capability gap Benchmark the exact model you plan to deploy, not just the flagship the paper reports; set a step/time ceiling per problem class informed by your own model's plateau, not the paper's K=20 default
Single-writer lock is a scaling ceiling Writer Exclusivity (A1) is a strict readers-writer lock; the paper's current deployment is one STRATUS instance per namespace with no concurrency control implemented Shard by namespace/system as the paper does, or implement the Sec 3.3 resource-set admission controller (Admit(Pi) <=> R(Pi) ∩ Rlocked = ∅) before running multiple concurrent mitigations against shared resources
RCA scores understate diagnosis quality AIOpsLab's RCA oracle grades against non-mutually-exclusive category labels; the paper documents a case where STRATUS's answer is arguably more precise than the oracle's expected label but is scored wrong regardless Treat AIOpsLab RCA percentages as a noisy lower bound, not a diagnosis-accuracy ground truth; spot-check disagreements before concluding a diagnosis is actually wrong
Confinement is necessary, not sufficient The paper states plainly that its rule-based confinement (Table 5) "may not be perfect to capture all destructive actions" Layer independent guardrails, a policy engine or risk-tiered approval, in front of the mitigation agent rather than relying on STRATUS's linter as the only control on irreversible actions

References

  • Chen, Pan, Clark, Su, Zheutlin, Bhavya, Arora, Deng, Jha, Xu, STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds (NeurIPS 2025, arXiv 2506.02009): https://arxiv.org/abs/2506.02009
  • STRATUS repository (UIUC xlab, verified public 2026-07-16): https://github.com/xlab-uiuc/stratus
  • Chen, Shetty, Somashekar, Ma, Simmhan, Mace, Bansal, Wang, Rajmohan, AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds (MLSys 2025, arXiv 2501.06706): https://arxiv.org/abs/2501.06706
  • Jha, Arora, Watanabe, Yanagawa, Chen, Clark, Bhavya, et al., ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks (ICML 2025, arXiv 2502.05352): https://arxiv.org/abs/2502.05352
  • CrewAI (multi-agent framework STRATUS is implemented on): https://www.crewai.com/
  • Alpern and Schneider, Recognizing Safety and Liveness (Distributed Computing, 1987), the formal safety-property class TNR instantiates: https://doi.org/10.1007/BF01782772

Related: AIOpsLab · Agentic incident management: OpsAgent · Agentic AIOps · Risk-tiered human approval gates · Multi-agent collaboration · Agent policy engine · Agent intent verification · Governing self-modifying agents · Evaluation integrity and anti-gaming · Operational runbooks

  1. Sec 1 (Introduction) and the paper's contributions list: STRATUS differs from prior human-assist AIOps tools (data collection, summarization, root-cause prediction) by targeting autonomous mitigation of live systems; multi-agent design chosen for role specialization, customizability/extensibility, and easier agent-safety reasoning; deterministic state-machine control flow with LLM-driven data flow per agent; artifacts at https://github.com/xlab-uiuc/stratus. ↩

  2. Sec 2 (Background and Problem Definition): environment E with states S, crash state ⊥; severity μ(s) = w1|A| + w2|V| + w3|L| (alerts, SLA violations, unhealthy-node capacity loss), wi > 0, μ(⊥) = ∞; four SRE tasks (detection, localization, RCA, mitigation); RCA explicitly stated as not on the mitigation critical path and typically done offline. ↩

  3. Sec 3 (STRATUS Design), first two paragraphs: agent set A = {αD, αG, αM, αU} with responsibilities (αD initial error state; αG localization+RCA; αM plan/decompose/execute mitigation; αU undo sequence on abort); action space Aread/Awrite/Aundo via ACI over kubectl; αD, αG restricted to Aread; state-transition relation R : S x Awrite -> S ∪ {⊥}. ↩

  4. Sec 3.1 (Safety Specification): assumptions A1 Writer Exclusivity (A-Lock, readers-writer lock over αM/αU), A2 Faithful Undo (U(spost) = spre exactly), A3 Bounded Risk Window (transaction length k, 1 <= k <= K); transaction rules R1 Checkpoint, R2 Execute, R3 Commit/Abort (commit iff spost != ⊥ and μ(spost) <= μ(spre), else abort via αU); Lemma 3.1 (induction proof that every externally visible state satisfies μ(s) <= b, the severity at initial detection); TNR framed as an Alpern-Schneider safety property (Sec 3.2). ↩↩

  5. Table 1 (the four worked mitigation plans): Node drain/rebalance (cordon, evict, scale), hidden mu-path 12 -> 18 -> 9, commit, visible 12 -> 9; Rolling upgrade (scale 0, patch, scale 3), hidden 15 -> 22 -> 11, commit, visible 15 -> 11; Bad image attempted (scale 0, patch(bad), scale 3), hidden 15 -> 24 -> 30, abort, visible 15 -> 15; Single hot-fix (K=1, apply hotfix), hidden 15 -> x, commit iff x <= 15 else abort, visible <= 15 either way. ↩

  6. Sec 3.3 (Extension): current deployment is one STRATUS instance per cloud system (separate namespace), no concurrency control between writer agents; described (not implemented) extension is a Coordination Controller / distributed lock manager over per-plan resource sets R(Pi), admission rule Admit(Pi) <=> R(Pi) ∩ Rlocked = ∅, releasing resources on transaction commit/abort. ↩↩

  7. Sec 4 (Implementation) and 4.1 (Realizing TNR): built on the CrewAI multi-agent framework, state-machine control flow (Fig 2), off-the-shelf LLMs (GPT, Llama) with no fine-tuning; Writer Exclusivity via role-based sandboxing (readers cannot mutate) plus state-machine mutual exclusion for writer agents, with kubectl --dry-run used where applicable; Faithful Undo via a stack-based rollback mechanism (Fig 3: push/pop of (command, resource_type, state) tuples, LIFO order), destructive irrecoverable actions rejected or rewritten to a recoverable form (file delete -> move to backup volume); explicit limitation that "our rule-based confinement may not be perfect to capture all destructive actions"; hybrid-operations note that writer agents can request human approval before execution with interceptable action stacks and tool interfaces. ↩↩↩↩↩↩↩

  8. Appendix A (Confinement Rules) and Table 5: read-only agents restricted to non-mutating commands (kubectl get/describe/logs, ls, cat, TelemetryCollection); writer agents restricted to sequential (serialized) execution; blocked for all agents: namespace deletion, kubectl debug/edit, stdin redirection (-f -), interactive flags (--stdin, --tty, exec -it), shell pipes, compound commands (&&, ||, ;), command substitution, flow control, shell function definitions; static linting validates syntax before execution; in evaluation, 10.5% of STRATUS (GPT-4o)'s commands and 9.7% of STRATUS (Llama 3.3)'s would have caused system hangs absent confinement. ↩↩↩

  9. Sec 4.2 (Other Notable Implementations) and Appendix D-E: observability tools (GetLogsTool, GetTracesTool/ReadTracesTool, GetBootstrapAnalysisTool/ReadBootstrapAnalysisTool, OracleTools) and command-line tools (NL2Kubectl, LintingTool, RollbackTool, NotificationTool); direct raw-metric consumption disabled, information surfaced via pre-processed logs/summaries instead; trace-based bootstrapping builds a call graph to seed the diagnosis agent's initial fault-localization hypothesis; termination combines three "weak" oracles (alert clearance, user-request success/no 4xx-5xx, system health) into one pass/fail gate, illustrated in Appendix E's wrong_bin_usage-mitigation-1 example (first oracle check: 115/117 failed responses despite healthy pods; second check after patch: 0/120 failed). ↩↩↩

  10. Sec 5 (Evaluation), opening paragraphs: benchmarks AIOpsLab and ITBench, each a live emulated-cloud arena with per-problem oracles; baselines AOL-agent and ITB-agent (each suite's reference agent), plus ReAct and Flash from AIOpsLab; models GPT-4o (gpt-4o-2024-08-06), GPT-4o-mini (gpt-4o-mini-2024-07-18), Llama 3.3 (llama-3-3-70b-instruct), temperature 0; metrics success rate, time, steps, USD cost; each benchmark took STRATUS 15-20 machine hours to complete. ↩↩↩

  11. Table 2 (mitigation results): AIOpsLab, 13 problems, ReAct(4o) 23.1%/46.0s/23.0 steps/$0.112, Flash(4o) 38.5%/154.0s/23.1/$0.150, AOL-agent(4o) 46.2%/223.3s/21.7/$0.206, AOL-agent(mini) 7.7%/58.9s/22.7/$0.003, AOL-agent(llama) 15.4%/98.2s/13.0/$0.037, STRATUS(4o) 69.2%/811.9s/46.3/$0.877, STRATUS(mini) 23.1%/3557.9s/125.7/$0.036, STRATUS(llama) 23.1%/1486.9s/71.8/$0.360; ITBench, 18 problems, ITB-agent(4o) 9.2%/251.7s, ITB-agent(llama) 5.7%/440.8s, STRATUS(4o) 50.0%/1720.8s/115.7/$6.11, STRATUS(mini) 19.4%/3874.9s/468.9/$9.38, STRATUS(llama) 28.0%/2566.6s/160.3/$0.76; text states 1.5x (AIOpsLab) and 5.4x (ITBench) improvement over second-best. ↩↩↩

  12. Table 3 (AIOpsLab ablation, GPT-4o): STRATUS 69.2%/811.9s/$0.877; No retry 15.4%/72.6s/$0.163; Naive retry w/o undo 23.1%/1221.5s/$0.929. Table 6 (ITBench ablation, GPT-4o): STRATUS 50%/1720.8s/$6.11; No Retry 11.1%/267.2s/$0.72; Naive retry w/o undo 50%/1711.9s/$5.13 (same task-level success because 8/18 problems are solved by pod-restart alone, independent of undo). Fig 5: retry-count PDF, limit of 9 retries, 80+% of problems retry at least once, 30+% retry 5 or more times. Sec 5 prose: in the 8 ITBench problems, restarting failing pods one-by-one clears alerts because the fault injector cannot recognize the original pod; among issued mitigation actions in those 8, ~15.2% are pod deletions auto-recreated by the Deployment controller, ~44.2% are hallucinated/wrong values (e.g. a nonexistent image name) fixed on the next retry via the reflection mechanism. ↩↩↩↩

  13. Table 4 (AIOpsLab, other SRE tasks, 86 problems total: 32 detection, 28 localization, 26 RCA): STRATUS(4o) Detection 90.6%/48.4s/$0.118, Localization 51.2%/65.3s/$0.126, RCA 34.6%/39.6s/$0.068; STRATUS(mini) 78.1%/25.0%/30.8%; STRATUS(llama) 93.8%/36.3%/26.9%; baselines ReAct(4o) 87.5%/26.8%/23.1%, Flash(4o) 59.4%/39.3%/26.9%, AOL-agent(4o) 62.5%/46.9%/38.5%. Localization scored Pass@3. RCA-label-ambiguity example: a k8s target-port misconfiguration's oracle expects layer "Virtualization" and fault type "Misconfiguration"; STRATUS answers layer "Application" and fault type "Network/Storage Issue", scored incorrect despite the paper arguing STRATUS's answer is more precise (no hypervisor-style virtualization layer exists in AIOpsLab). ↩

  14. Appendix B (Bounded Risk Window) and Fig 6: step-limit sweep over {3, 5, 10, 15, 20, 30} on AIOpsLab across all four task types; STRATUS (GPT-4o) improves from 0% to 53.53% success moving from a 3-step to a 10-step limit; performance plateaus past 15 steps across evaluated agents; K=20 set as the default from this analysis. ↩