STRATUS: transactional non-regression for autonomous SRE¶
Scope: STRATUS (arXiv 2506.02009, NeurIPS 2025), a multi-agent LLM system that mitigates live cloud failures without a human in the loop, built around a formalized safety property, Transactional Non-Regression (TNR), that bounds how much worse an autonomous mitigation attempt can make the system before an undo restores the prior state. Covers the environment and severity model, the TNR assumptions and transaction semantics with proof sketch, the CrewAI-based state-machine implementation (confinement, stack-based undo, oracles), and the AIOpsLab/ITBench results including the undo-and-retry ablation. The benchmark suites it runs against are AIOpsLab and, by name only in this page (not separately covered in this KB), ITBench; the pipeline-shaped sibling in this KB is OpsAgent; the field view of guardrails and telemetry compaction for agentic ops is agentic AIOps; the general oversight-ladder pattern STRATUS's "hybrid operations" hook could plug into is risk-tiered approval gates.
Independently verified here: the full paper text (PDF via arXiv, extracted with
pdftotext -layout), all tables and figures cited below, and the code repositoryhttps://github.com/xlab-uiuc/stratus, cloned at commit4fc9a5b66bebbc5ddf2154c06e97f05db9385183(2025-10-23, Apache 2.0), not merely confirmed public. Every file quoted in "How to bring up the released repository" below (.env.tmpl,pyproject.toml,src/stratus/main.py,src/stratus/agent/config.py,src/stratus/agent/aiopslab.py,src/stratus/action_stack.py,src/stratus/tools/mitigation/rollback_tool.py,src/stratus/tools/linting/kubectl_linter.py,src/stratus/tools/kubectl/kubectl.py,Dockerfile,test_agent.sh,eval/eval.py,eval/eval_tasks.yaml) is the real source, read directly from the clone, not paraphrased.KubectlLinter(), the class this page originally assumed was the paper's Table 5 confinement layer, was instantiated directly in this sandbox and raised a real, reproducibleAttributeError(ResourceTypehas no attributeREPLICASET, referenced in theSCALEaction's valid-resource set but never defined in theResourceTypeenum) at this commit; this is documented below as a genuine repository defect, not a paper claim. No AIOpsLab cluster, LLM API key, orkindcluster was available in this sandbox, so no STRATUS mitigation episode was run end to end; the Python block below still only models the TNR transaction mechanism and reproduces Table 1's four worked examples numerically, and does not run STRATUS, CrewAI, or any LLM. Note a wording quirk in the paper itself: the abstract and introduction (Sec 1) say "Transactional No-Regression"; Section 3 and the contributions list say "Transactional Non-Regression"; both refer to the same TNR property, and this page uses "Non-Regression" since that is what the formal treatment (Sec 3) uses throughout.
What it is¶
STRATUS is an LLM-based multi-agent system that performs the full SRE lifecycle, detection, localization, root-cause analysis (RCA), and mitigation, against live emulated cloud services, with the explicit goal of acting autonomously rather than assisting a human operator.1 It models a target cloud as an environment E with a state set S, a crash state ⊥, and a severity metric μ(s) = w1|A| + w2|V| + w3|L| over alert count, SLA violations, and unhealthy-node capacity loss, with μ(⊥) = ∞.2 Four agents divide the work: a Detection agent αD establishes the initial error state; a Diagnosis agent αG handles both localization and RCA; a Mitigation agent αM plans and executes state-changing commands; and an Undo agent αU reverts a mitigation attempt if it fails. Detection and diagnosis are read-only (Aread); only mitigation and undo can mutate the system (Awrite, Aundo), issued through an Agent-Computer Interface (ACI) built on kubectl.3
The paper's central contribution is not the four-agent split itself but a safety specification for it: Transactional Non-Regression (TNR). TNR wraps every mitigation attempt in transaction semantics, checkpoint the entry state, execute a bounded sequence of commands, then commit only if the system did not crash and its severity did not increase versus the checkpoint, otherwise abort and undo, restoring the checkpoint exactly. The paper proves (Lemma 3.1, by induction) that under this discipline every externally visible state satisfies μ(s) ≤ b, the severity at the moment the failure was first detected, no matter how bad the system gets inside an aborted transaction's hidden execution path.4 This is what licenses STRATUS's headline mechanism: safe retry. Because a failed mitigation attempt is invisible to the rest of the system, STRATUS can try an ambitious plan, watch it fail, undo it for free, and try something else, up to nine times per problem.12
Why use it¶
- The safety property is the deliverable, not the agent count. TNR is what makes "let an LLM run mutating
kubectlcommands against production with no human approval" a defensible sentence: the worst case is bounded by the state the system was already in, formally, not just empirically.4 - Undo-and-retry is measurably the source of STRATUS's advantage, not multi-agent framing alone. Ablating retry entirely on AIOpsLab (13 mitigation problems, GPT-4o) drops success from 69.2% to 15.4%; keeping retry but removing undo (retrying from wherever the failed attempt left the system, instead of rolling back) only reaches 23.1%, worse time (1221.5s vs 811.9s) and worse cost ($0.929 vs $0.877) for a worse outcome. The paper's reading: failed mitigations that are not undone make the system harder to save on the next attempt.12
- It beats the field's reference agents by at least 1.5x on mitigation, the task that actually requires safety. STRATUS (GPT-4o) solves 69.2% of AIOpsLab's 13 mitigation problems versus 46.2% for the next-best baseline (AOL-agent), and 50.0% of ITBench's 18 versus 9.2% (ITB-agent); the smallest ratio across all model/benchmark pairs in Table 2 is 1.5x (both AIOpsLab GPT-4o and AIOpsLab Llama 3.3), consistent with the abstract's "at least 1.5 times" claim.11
- The confinement layer prevents concrete hangs, not just abstract risk. Interactive and stdin-blocking
kubectlinvocations (exec -it,apply -f -) would have hung the agent session; the paper measured that 10.5% of the commands STRATUS (GPT-4o) issued, and 9.7% of STRATUS (Llama 3.3)'s, would have caused exactly this failure mode absent the confinement rules.8 - The cost is honestly reported, and it is real. Safety buys retries, and retries cost tokens and wall-clock time: STRATUS (GPT-4o) takes 3.6x the time (811.9s vs 223.3s) and 4.3x the cost ($0.877 vs $0.206) of AOL-agent on the same AIOpsLab mitigation set. The paper frames this as still cheap relative to human SRE time, without benchmarking that comparison directly.11
When to use it (and when not)¶
- Use it as the reference design for autonomous mitigation against declarative, reconciliation-based platforms (Kubernetes, and by the paper's own analogy Borg/Twine/ECS-style systems), where "faithful undo" is tractable because the platform already tracks and can restore prior object state.7
- Use it where the cost of an autonomous action making things measurably worse is the actual blocker to deployment, and a formally bounded worst case (never below the pre-incident baseline, in the externally visible trace) is worth paying 3-4x the latency and cost of a baseline agent that, per the paper, lacks TNR's safe-retry capability.
- Do not treat ITBench's headline mitigation number as proof of robust fixes. In 8 of ITBench's 18 mitigation problems, STRATUS wins because the benchmark's fault injector cannot re-target a pod after it restarts, so repeatedly restarting failing pods clears the alert without addressing the injected fault; the paper states plainly this strategy would not work against persistent faults like misconfigurations or hardware defects.12
- Do not deploy it expecting concurrent multi-writer safety out of the box. TNR's Writer Exclusivity (A1) is a single readers-writer lock; the current deployment model is one STRATUS instance per cloud system, one namespace at a time. The paper's resource-aware Coordination Controller for concurrent writers is a described extension, not an implemented one.6
- Do not trust the RCA success numbers as a diagnosis-quality signal without inspection. AIOpsLab's RCA oracle uses non-mutually-exclusive category labels; the paper documents a case where STRATUS's answer ("Application" layer, "Network/Storage Issue" fault type) is arguably more precise than the oracle's expected answer ("Virtualization", "Misconfiguration") but is scored wrong regardless.13
- Do not assume the rule-based confinement layer catches every destructive action; the authors state it directly as a limitation, not a completeness guarantee.7
Architecture¶
flowchart TB
FAIL["Failure in cloud E<br/>baseline severity b = mu(se0)"] --> DET["Detection agent aD (Aread only)<br/>establishes initial error state se0"]
DET --> DIAG["Diagnosis agent aG (Aread only)<br/>localization + RCA, trace-based bootstrapping"]
DIAG --> CKPT["Mitigation agent aM acquires A-Lock<br/>R1: checkpoint spre"]
CKPT --> EXEC["R2: execute a1..ak, k less-or-equal K=20<br/>via NL2Kubectl + LintingTool + kubectl --dry-run"]
EXEC --> CLOUD["Cloud E (kubectl ACI, confinement-filtered)"]
CLOUD --> ORACLE{"R3: oracle check on spost<br/>alert clearance / user requests / system health"}
ORACLE -->|"commit: spost != crash AND mu(spost) less-or-equal mu(spre)"| VIS["externally visible state advances to spost"]
ORACLE -->|"abort"| UNDO["Undo agent aU acquires A-Lock<br/>Faithful Undo: U(spost) = spre via action stack"]
UNDO --> VISU["externally visible state restored to spre"]
VIS -->|"oracles fail: still unhealthy"| CKPT
VISU -->|"retry, up to 9x per problem"| CKPT
VIS -->|"all oracles pass"| DONE["NotificationTool: mitigation succeeded"]Writer Exclusivity (A1) is enforced two ways: role-based sandboxing (detection and diagnosis agents cannot issue mutating commands at all) and state-machine scheduling that serializes the writer agents (αM, αU), so at most one is active at a time, a readers-writer lock over the whole system.7 Faithful Undo (A2) is realized as a stack-based rollback: every mutating command pushes a (command, resource_type, prior_state) tuple, and undo mechanically pops the stack in LIFO order, restoring each affected Kubernetes object; irrecoverable actions (an outright file delete) are either rejected or rewritten into a recoverable equivalent (move to a backup volume instead).7 Bounded Risk Window (A3) caps a single transaction at K=20 commands, set from an empirical sweep (3, 5, 10, 15, 20, 30) where STRATUS (GPT-4o) jumps from 0% to 53.53% success moving from K=3 to K=10 and plateaus past 15.14
How to use it¶
STRATUS is not a library you import into an existing pipeline; it is a reference agent implementation over CrewAI, evaluated against two benchmark harnesses (AIOpsLab, ITBench) that already provide the live Kubernetes environments, fault injectors, and oracles.10 The paper's own evaluation runtime is a useful sizing reference: each of the two benchmark suites took STRATUS 15-20 machine hours to finish end to end.10 The exact bring-up commands, configuration schema, and adapter code are in "How to bring up the released repository" below.
Two implementation choices matter operationally. First, STRATUS disables direct raw-metric consumption by the agents; numerical telemetry is pre-processed into filtered, aggregated, human-readable summaries before it reaches the model, the same telemetry-compaction discipline covered in agentic AIOps.9 Second, termination is a combined-oracle decision, not a single check: alert clearance, absence of 4xx/5xx user-facing errors, and Kubernetes-level system health (pods, volumes) are each "weak oracles" individually, and STRATUS only concludes success when all three pass, illustrated in the paper by a case where the cluster looked healthy at the orchestration level while the workload oracle still measured 115 failed responses out of 117.9
How to bring up the released repository¶
Clone, submodule, and install¶
git clone https://github.com/xlab-uiuc/stratus.git
cd stratus
git checkout 4fc9a5b66bebbc5ddf2154c06e97f05db9385183 # 2025-10-23, Apache 2.0
# AIOpsLab is a git submodule, not a PyPI dependency (.gitmodules):
# [submodule "AIOpsLab"] path = AIOpsLab url = https://github.com/xlab-uiuc/AIOpsLab/
git submodule update --init --recursive
python3.12 -m venv .venv && source .venv/bin/activate
pip install uv crewai crewai-tools
crewai install # creates/refreshes .venv from pyproject.toml + uv.lock
cp .env.tmpl .env # then fill in PROVIDER_AGENTS/MODEL_AGENTS/API_KEY_AGENTS etc.
pyproject.toml pins crewai==0.95.0, crewai-tools>=0.25.8, kubernetes==30.1.0, litellm==1.65.0, bashlex>=0.18, and 25 more; [project.scripts] registers four real console entry points backed by src/stratus/main.py: stratus/run_crew (main:run), train (main:train), replay (main:replay), test (main:test).
The real configuration surface¶
.env.tmpl (copy to .env) is the actual, complete environment-variable contract, not a paraphrase: separate LLM backend blocks for agents versus tools (PROVIDER_AGENTS/MODEL_AGENTS/URL_AGENTS/API_KEY_AGENTS/REASONING_EFFORT_AGENTS/SEED_AGENTS/TOP_P_AGENTS/TEMPERATURE_AGENTS/THINKING_AGENTS/THINKING_BUDGET_AGENTS/MAX_TOKENS_AGENTS, mirrored as *_TOOLS), a service block (GOD_MODE which "will disable safeguards during mitigation step" if set "True", KUBECONFIG, GRAFANA_URL, TOPOLOGY_URL), and an evaluation block (BENCHMARK = AIOpsLab or ITBench, AGENT_TASK_DIRECTORY, SRE_AGENT_EVALUATION_DIRECTORY, INCIDENT_NUMBER, EXP_NAME). src/stratus/agent/config.py's StratusAgentConfig (a real Pydantic BaseModel, quoted directly) is the schema every run actually validates against:
class StratusAgentConfig(BaseModel):
run_mode: str = Field(default="VALIDATION_RETRY") # NAIVE | VALIDATION_RETRY | BLINDLY_RETRY
max_retry_attempts: int = Field(default=10)
retry_wait_time: float = Field(default=60)
validation_wait_time: float = Field(default=30)
output_dir: str = Field(default="./stratus_output")
namespace: str = Field(default="")
dropout_threshold: int = Field(default=4)
use_dry_run: bool = Field(default=False) # "Warning: Dry run is not well-implemented yet."
verify_dry_run: bool = Field(default=False)
forbid_unsafe_commands: bool = Field(default=False)
use_rollback_stack: bool = Field(default=True)
validate_rollback: bool = Field(default=False)
clear_replicaset: bool = Field(default=True) # "Warning: may be harmful to the system."
clear_rs_wait_time: float = Field(default=5)
config.validate() enforces real invariants: run_mode must be one of the three literals above; max_retry_attempts must be 1 whenever run_mode != "VALIDATION_RETRY"; output_dir is created if missing and checked writable, raising PermissionError otherwise. retrieve_config_from_env() builds this object from the environment (RUN_MODE, MAX_RETRY_ATTEMPTS, SLEEP_BETWEEN_RETRIES, VALIDATION_WAIT_TIME, OUTPUT_DIRECTORY), the ITBench-path entry used by main.run().
The real AIOpsLab adapter and CrewAI role config¶
src/stratus/main.py's run() branches on BENCHMARK. For AIOpsLab, the entry point is genuinely simple, three real objects, not a "wire it yourself" gap:
from aiopslab.orchestrator import Orchestrator
from stratus.agent.aiopslab import StratusAgent_AIOpsLab
orchestrator = Orchestrator()
problem_desc, _instructs, _apis = orchestrator.init_problem(problem_id) # e.g. "k8s_target_port-misconfig-mitigation-1"
agent = StratusAgent_AIOpsLab(problem_desc=problem_desc, task_type=task_type, extra_oracles=wrk_oracle)
orchestrator.register_agent(agent, name="StratusAgent_AIOpsLab")
agent.run()
asyncio.run(orchestrator.start_problem(max_steps=30))
agent.finalize()
StratusAgent_AIOpsLab (src/stratus/agent/aiopslab.py) is a real Pydantic BaseModel subclass of StratusAgentBase, threaded around a Generator[str, Any, NoReturn] communicator with prompt_semaphore/command_semaphore so the synchronous CrewAI crew and AIOpsLab's async orchestrator can hand control back and forth per step; it extracts the target namespace straight from the problem description (extract_kubernetes_namespace) and assigns it to self.config.namespace before the crew starts. src/stratus/config/agents-aiopslab.yaml is the real, shipped CrewAI role definition for three agents (sre_diagnosis_agent, sre_mitigation_agent, sre_rollback_agent); their goal: prompts are explicit about tool-usage discipline the paper's prose only summarizes, e.g. "the interactive operations like kubectl edit are not supported by NL2Kubectl tool. You should use kubectl apply instead," and "you should get all the pods and deployments to figure out what kind of services are running in the cluster" as the diagnosis agent's mandated first move.
The real Faithful Undo mechanism (A2)¶
src/stratus/action_stack.py's ActionStack is a thread-safe singleton (__new__ with a class-level threading.Lock) wrapping a plain Python list of RollbackNode objects; push/pop/peek/clear are the entire real interface, each logging through the standard logging module. src/stratus/tools/mitigation/rollback_tool.py defines the two real Pydantic dataclasses actually pushed onto it:
@dataclass
class RollbackCommand:
command_type: str # "command" or "file"
content: str
@dataclass
class RollbackNode:
action: str
rollback: list[RollbackCommand]
cluster_state: str | None = None
RollbackTool._run() (a real CrewAI BaseTool) pops one RollbackNode and replays its rollback list: a "command" entry runs KubeCtl.exec_command(content) directly; a "file" entry calls _restore_cluster_state, which parses a captured YAML dump (kubectl get all -o yaml, saved before the mitigation attempt), splits it into CustomResourceDefinitions first, then applies remaining resources in four tiers (Namespace/ConfigMap/Secret/ServiceAccount/Role/RoleBinding, then Service/PVC/PV, then DaemonSet/Job/CronJob, then Deployment/StatefulSet last, with an optional clear_replicaset pass afterward). If config.validate_rollback is set, it re-captures cluster state after the restore and writes a real unified diff (difflib.unified_diff) to <output_dir>/rollback_validation/rollback_diff_<name> for post-hoc audit, exactly the "rollback-state serialization" artifact this page previously only asserted existed.
The real confinement layer, and a genuine defect in it¶
Re-verified 2026-07-17 against a fresh clone at the same pinned commit, with the repository's own source read directly rather than trusting an earlier pass of this page. The paper's Table 5 (blocked namespace deletion, interactive flags, stdin redirection, pipes, compound commands, command substitution) is not implemented as a standalone rule list in this repository; three code paths bear on it, and they are not equally real. Ranked by what is actually wired into the mitigation path:
First, and doing most of the real work: NL2KubectlCustomTool._check_kubectl_command (src/stratus/tools/kubectl/nl2kubectl.py), called unconditionally at the top of every _run() before a command reaches the cluster. It outright rejects kubectl debug, edit, wait, proxy, port-forward, and cp by prefix match, raising before execution. It then parses the command with bashlex.parse and walks the AST: any node whose kind is outside {command, heredoc, redirect, tilde, word} raises ("Pipe commands are forbidden" for pipe nodes, "Unsupported operator kind: <kind>" for everything else, which is what actually rejects ;/&&/|| compound commands and command substitution, since bashlex represents those as distinct node kinds); write redirection (>) raises "Write redirection is forbidden."; and a second, separate token scan rejects --interactive, -i, --tty, -t, --stdin, and -it outright. This single function covers most of Table 5's list directly, client-side, before any API call.
Second: KubeCtl.dry_run_json_output (src/stratus/tools/kubectl/kubectl.py) runs whatever survives the first layer through kubectl ... --dry-run=server and inspects stderr: the literal substring "can't be used with attached containers options" maps to DryRunStatus.ERROR, "Interactive command is not supported.", which is how exec -it is actually rejected (kubectl exec itself is not in the blocked-subcommand list, only its interactive flags are caught, and only by the first layer's flag scan or, if that were absent, here).
A genuine gap, independently confirmed in this sandbox against a real cluster, not in either layer above: kubectl exec does not support --dry-run at all.
$ kubectl -n hotel-res exec dry-run-exec-test --dry-run=server -- echo hi
error: unknown flag: --dry-run
See 'kubectl exec --help' for usage.
dry_run_json_output maps that exact stderr string to DryRunStatus.NOEFFECT ("Dry-run not supported. Possibly it's a safe command."), and the NOEFFECT branch in NL2KubectlCustomTool._run() executes the command immediately, no preview, no rollback plan pushed to the action stack. A non-interactive kubectl exec -n ns pod -- <command> (no -it, so it clears the first layer's flag scan) therefore runs for real with no dry-run gate at all. This is a real confinement gap distinct from the linter defect below, and the RBAC manifest further down does not grant the pods/exec subresource, specifically to close it at the authorization layer since the tool layer does not.
Third, and unreachable in practice: src/stratus/tools/linting/kubectl_linter.py's KubectlLinter validates only that a given KubectlAction (get, delete, describe, logs, exec, port-forward, apply, create, edit, scale) is legal for a given ResourceType; it does not block namespace deletion, pipes, redirection, or command substitution at all, so even a working copy would not be the Table-5 confinement layer this page originally assumed it was. It also fails to construct, confirmed by running it directly against the pinned commit, freshly re-cloned this session:
$ python3 -c "from stratus.tools.linting.kubectl_linter import KubectlLinter; KubectlLinter()"
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "src/stratus/tools/linting/kubectl_linter.py", line 58, in __init__
KubectlAction.SCALE: {ResourceType.DEPLOYMENT, ResourceType.STATEFULSET, ResourceType.REPLICASET},
^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: type object 'ResourceType' has no attribute 'REPLICASET'
ResourceType (defined a few lines above in the same file) has no REPLICASET member, so KubectlLinter.__init__ cannot run and KubectlLinter() cannot be constructed at all at this commit. More importantly, and not previously checked on this page: grep -rn "KubectlLinter" src at this commit returns exactly one match, the class's own definition; nothing else in the repository imports or instantiates it. Its siblings PromQLLinter, LogQLLinter, and JaegerLinter (same directory) are genuinely wired into nl2metrics.py/nl2logs.py/nl2traces.py; KubectlLinter has no analogous caller anywhere. The AttributeError is real and reproducible, but it is dead code failing, not a live crash risk in the shipped mitigation path: nothing calls KubectlLinter() today, so it never executes. The paper's Table 5 completeness claim rests entirely on the first two mechanisms above, which are real, wired-in, and independently verified here, not on this class. File the REPLICASET gap upstream if you plan to wire the linter in yourself, but do not expect fixing it alone to change the confinement posture of the released repository: the AST check and the dry-run gate are what is actually protecting a live cluster today, and the kubectl exec gap above is the more consequential open issue.
Real cluster bring-up and the evaluation entry point¶
test_agent.sh is the actual per-task runner and the closest thing to an AIOpsLab bring-up script in the repository:
kind delete cluster --name kind
kind create cluster --config ./AIOpsLab/kind/kind-config-x86.yaml # or kind-config-arm.yaml
export TASK_NAME=k8s_target_port-misconfig-mitigation-1
export KUBECONFIG=$HOME/.kube/config
export PYTHONPATH=$(pwd):$(pwd)/AIOpsLab:$(pwd)/src:$PYTHONPATH
export OUTPUT_DIRECTORY=eval/$(date +%m-%d_%H-%M-%S)-${TASK_NAME}/stratus_output
crewai reset-memories -a
crewai run
bash test_agent.sh -r x86 k8s_target_port-misconfig-mitigation-1 wraps exactly this. eval/eval.py reads eval/eval_tasks.yaml, a real, shipped list of task IDs grouped by detection/localization/analysis/mitigation (32 detection tasks, 13 mitigation tasks including k8s_target_port-misconfig-mitigation-1 through -3, auth_miss_mongodb-mitigation-1, redeploy_without_PV-mitigation-1, and others), and calls test_agent.sh once per task, exactly the harness eval/eval.py needs to run the paper's own AIOpsLab suite. No least-privilege Kubernetes RBAC manifest (ServiceAccount/Role/RoleBinding) ships in this repository; test_agent.sh exports KUBECONFIG=$HOME/.kube/config, so STRATUS's write access in this reference setup is whatever that kubeconfig already grants, cluster-admin on a fresh kind cluster by default. The paper's Writer Exclusivity property (A1) is enforced in application logic (role-based agent sandboxing plus a serialized writer lock), not by the Kubernetes RBAC layer, so RBAC scoping is entirely the operator's responsibility; the manifest below closes that gap concretely rather than only naming it.
A real, least-privilege RBAC manifest (server-validated, not just described)¶
Neither the repository nor an earlier revision of this page shipped an actual ServiceAccount/Role/RoleBinding. Re-verified 2026-07-17 against a fresh kind cluster (kind create cluster --name stratus-rbac-verify, kind v0.24.0, node image kindest/node:v1.31.0): applying the manifest an earlier pass of this page shipped surfaced two real defects that had not been checked, not just re-confirmed the happy path.
Bug 1, caught by kubectl auth can-i, not by reading the YAML: that earlier manifest granted get/list on nodes inside a namespaced Role. nodes is cluster-scoped (kubectl api-resources --api-group="" -o wide reports NAMESPACED false for nodes, true for pods/configmaps/services/events/resourcequotas), and a namespaced Role/RoleBinding can never authorize a cluster-scoped resource, no matter what verbs are listed; the rule is silently inert. Confirmed empirically: with that Role applied, kubectl auth can-i get nodes --as=<the service account> answered no. This matters here specifically because STRATUS's severity metric mu(s) has a node-health term (L, unhealthy-node capacity loss); an operator relying on that Role would believe node visibility was granted when it was not. The fix is a separate ClusterRole/ClusterRoleBinding, read-only, bound only to this ServiceAccount.
Bug 2, found by reading rollback_tool.py directly, not by inspecting the Role: RollbackTool._restore_cluster_state's _clear_replicasets (src/stratus/tools/mitigation/rollback_tool.py:181-209), which runs when config.clear_replicaset is true (the default), issues a real kubectl delete rs <names> -n <namespace>. The earlier Role granted only get/list/watch/patch/update on replicasets, no delete, which would have made the default Faithful Undo path fail with a real Forbidden at the exact moment a rollback needed it.
# rbac.yaml -- scoped to one target namespace, matching a single-mitigation-target
# deployment (Writer Exclusivity, A1, is one STRATUS instance per managed system).
apiVersion: v1
kind: ServiceAccount
metadata:
name: stratus-agent
namespace: hotel-res
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: stratus-mitigator
namespace: hotel-res
rules:
- apiGroups: ["apps"]
resources: ["deployments", "statefulsets"]
verbs: ["get", "list", "watch", "patch", "update"]
- apiGroups: ["apps"]
resources: ["replicasets"]
verbs: ["get", "list", "watch", "patch", "update", "delete"] # delete: rollback_tool.py's clear_replicaset path
- apiGroups: [""]
resources: ["pods", "pods/log", "configmaps", "services", "events"]
verbs: ["get", "list", "watch"]
- apiGroups: [""]
resources: ["pods"]
verbs: ["delete"] # restart-based mitigation and rollback's evict/recreate path
# pods/exec is deliberately NOT granted: kubectl exec has no --dry-run
# support (verified above), so RBAC is the only gate left for it.
- apiGroups: [""]
resources: ["resourcequotas"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: stratus-mitigator-binding
namespace: hotel-res
subjects:
- kind: ServiceAccount
name: stratus-agent
namespace: hotel-res
roleRef:
kind: Role
name: stratus-mitigator
apiGroup: rbac.authorization.k8s.io
---
# nodes is cluster-scoped (verified above); a namespaced Role can never
# authorize it, so the severity metric's node-health term needs a separate,
# read-only ClusterRole bound only to this ServiceAccount.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: stratus-node-reader
rules:
- apiGroups: [""]
resources: ["nodes"]
verbs: ["get", "list"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: stratus-node-reader-binding
subjects:
- kind: ServiceAccount
name: stratus-agent
namespace: hotel-res
roleRef:
kind: ClusterRole
name: stratus-node-reader
apiGroup: rbac.authorization.k8s.io
$ kubectl apply --dry-run=server -f rbac.yaml
serviceaccount/stratus-agent created (server dry run)
role.rbac.authorization.k8s.io/stratus-mitigator created (server dry run)
rolebinding.rbac.authorization.k8s.io/stratus-mitigator-binding created (server dry run)
clusterrole.rbac.authorization.k8s.io/stratus-node-reader created (server dry run)
clusterrolebinding.rbac.authorization.k8s.io/stratus-node-reader-binding created (server dry run)
$ kubectl apply -f rbac.yaml # applied for real, then checked the actual boundary
serviceaccount/stratus-agent created
role.rbac.authorization.k8s.io/stratus-mitigator created
rolebinding.rbac.authorization.k8s.io/stratus-mitigator-binding created
clusterrole.rbac.authorization.k8s.io/stratus-node-reader created
clusterrolebinding.rbac.authorization.k8s.io/stratus-node-reader-binding created
$ SA=system:serviceaccount:hotel-res:stratus-agent
$ kubectl auth can-i get pods -n hotel-res --as=$SA
yes
$ kubectl auth can-i patch deployments -n hotel-res --as=$SA
yes
$ kubectl auth can-i delete pods -n hotel-res --as=$SA
yes
$ kubectl auth can-i delete replicasets -n hotel-res --as=$SA
yes
$ kubectl auth can-i get nodes --as=$SA
yes
$ kubectl auth can-i patch nodes --as=$SA
no
$ kubectl auth can-i create pods/exec -n hotel-res --as=$SA
no
$ kubectl auth can-i get secrets -n hotel-res --as=$SA
no
$ kubectl auth can-i delete namespaces --as=$SA
no
$ kubectl auth can-i list pods --all-namespaces --as=$SA
no
$ kubectl auth can-i delete deployments -n hotel-res --as=$SA
no
$ kubectl auth can-i create deployments -n hotel-res --as=$SA
no
The service account can read and mitigate (get/patch/delete on pods, deployments, statefulsets, replicasets) inside hotel-res, and can read node health cluster-wide through the separate ClusterRole, but cannot patch or delete nodes, cannot exec into a pod (closing the dry-run gap above at the authorization layer), cannot read secrets, cannot delete a namespace, cannot list pods outside its namespace, and cannot create or delete Deployments outright, since the paper's own worked examples (Table 1) only ever patch, scale, or evict existing objects, never create or delete a Deployment. This Role is scoped to that demonstrated task shape; a mitigation problem class that genuinely needs to create new objects (or reach ConfigMaps/Services with write verbs) requires deliberately widening it, not assuming NL2Kubectl's full theoretical command vocabulary (kubectl_unsafe_commands in nl2kubectl.py also lists autoscale, cordon, taint, uncordon, label, certificate, among others) is pre-authorized.
Workload shape: a Job, not a Deployment, and a sudo grant with no purpose to remove¶
The registered run_crew project entry point executes exactly one AIOpsLab task and exits (main.run() returns once orchestrator.start_problem() finishes); there is no server loop. A Kubernetes Deployment implies a process the platform should keep restarting indefinitely, which is the wrong shape for a workload that is supposed to terminate; a Job (parallelism: 1, completions: 1) is the honest fit and still satisfies Writer Exclusivity (A1: at most one writer instance active against the managed system). The first real cluster run on 2026-07-17 fixed the missing image command but exposed a second defect: CrewAI's crewai run wrapper swallowed the child process's nonzero exit, so Kubernetes marked a failed task complete. The manifest now invokes uv run run_crew directly, bypassing that wrapper; the failure-propagation regression below was run against the exact pinned source on 2026-07-18.
# job.yaml -- build the image locally from the pinned Dockerfile (CMD and
# sudo fixes below applied); no published image tag exists upstream at this commit.
apiVersion: batch/v1
kind: Job
metadata:
name: stratus-agent-hotel-res-mitigation-1
namespace: hotel-res
spec:
parallelism: 1
completions: 1
backoffLimit: 0 # Faithful Undo (A2) already handles a failed attempt; do not let Kubernetes retry the whole task
template:
metadata:
labels: { app: stratus-agent }
spec:
restartPolicy: Never
serviceAccountName: stratus-agent
automountServiceAccountToken: true
containers:
- name: stratus
image: stratus:local-build
command: ["uv", "run", "run_crew"] # direct project entry point; preserves a failing exit code
securityContext:
runAsGroup: 0 # vanilla Kubernetes needs this requested explicitly: the chgrp -R 0/chmod g=u
# pattern below only takes effect for free under OpenShift's SCC-injected GID 0
env:
- name: BENCHMARK
value: "AIOpsLab"
- name: TASK_NAME
value: "misconfig_app_hotel_res-mitigation-1"
envFrom:
- secretRef:
name: stratus-llm-credentials # PROVIDER_AGENTS/MODEL_AGENTS/URL_AGENTS/API_KEY_AGENTS, not committed here
resources:
requests: { cpu: "500m", memory: "512Mi" }
limits: { cpu: "2", memory: "2Gi" }
The following cluster transcript is the failing pre-fix reproduction, captured with command: ["crewai", "run"]. It proves why the wrapper is unsafe as a Job entry point; it is not presented as output from the corrected manifest above.
$ docker build -t stratus:local-build . # corrected Dockerfile: CMD added, sudo grant removed (below)
...
writing image sha256:5da34bdcf7177247ae788bf3246f76195a2436e20ee7c0b7cf19a09a0f54f3eb done
$ kind create cluster --name stratus-cmd-fix-verify
...
Set kubectl context to "kind-stratus-cmd-fix-verify"
$ kind load docker-image stratus:local-build --name stratus-cmd-fix-verify
Image: "stratus:local-build" with ID "sha256:5da34bdcf7177..." not yet present on node "stratus-cmd-fix-verify-control-plane", loading...
$ kubectl apply --dry-run=server -f job.yaml
job.batch/stratus-agent-hotel-res-mitigation-1 created (server dry run)
$ kubectl apply -f job.yaml # applied for real
job.batch/stratus-agent-hotel-res-mitigation-1 created
$ kubectl -n hotel-res get pods -l app=stratus-agent
NAME READY STATUS RESTARTS AGE
stratus-agent-hotel-res-mitigation-1-8mfqs 1/1 Running 0 13s
$ kubectl -n hotel-res logs stratus-agent-hotel-res-mitigation-1-8mfqs --tail=12
Installed 279 packages in 981ms
Traceback (most recent call last):
File "/app/stratus/.venv/bin/run_crew", line 4, in <module>
from stratus.main import run
File "/app/stratus/src/stratus/main.py", line 49, in <module>
from stratus.crew import StratusCrew
File "/app/stratus/src/stratus/crew.py", line 23, in <module>
from stratus.llm_backends.init_backend import get_llm_backend_for_agents, get_llm_backend_for_tools
File "/app/stratus/src/stratus/llm_backends/init_backend.py", line 56, in <module>
URL_AGENTS = os.environ["URL_AGENTS"].rstrip("/")
KeyError: 'URL_AGENTS'
An error occurred while running the crew: Command '['uv', 'run', 'run_crew']' returned non-zero exit status 1.
$ kubectl -n hotel-res get job stratus-agent-hotel-res-mitigation-1
NAME STATUS COMPLETIONS DURATION AGE
stratus-agent-hotel-res-mitigation-1 Complete 1/1 36s 40s
$ kubectl -n hotel-res get pod stratus-agent-hotel-res-mitigation-1-8mfqs -o jsonpath='{.status.containerStatuses[0].state.terminated.exitCode}'
0
CrewAI 0.95.0's crewai/cli/run_crew.py catches subprocess.CalledProcessError, prints it, and does not re-raise or call sys.exit. The smallest real regression runs both commands against the same pinned checkout and missing-credential failure:
crewai-wrapper: exit=0 KeyError: 'PROVIDER_AGENTS'
direct-entry: exit=1 KeyError: 'PROVIDER_AGENTS'
failure propagation: PASS
The harness uses subprocess.run() for .venv/bin/crewai run and uv run run_crew, asserts the first return code is 0, and asserts the second is nonzero. Both reach the same real stratus.llm_backends.init_backend import and the same missing environment boundary; only the wrapper changes. This locks down the corrected Job semantics: missing credentials, an import failure, or another unhandled application error now fails the container and therefore the Job instead of producing a false green.
The earlier cluster run still establishes that the Pod can be admitted under the scoped service account and execute the real dependency environment. It does not establish a successful mitigation. The corrected direct-entry Job was not rerun with real LLM credentials, so no mitigation or undo episode is claimed. Without runAsGroup: 0, the Pod also fails with PermissionError: [Errno 13] Permission denied: 'pyproject.toml', because the chgrp -R 0 /app && chmod -R g=u /app pattern below only benefits a process actually running under group 0, which OpenShift's SCCs inject automatically but vanilla Kubernetes does not.
The shipped Dockerfile also creates the non-root stratus-agent user (UID 1001) and then grants it passwordless sudo over everything:
RUN useradd -m -u 1001 -G sudo -s /bin/bash stratus-agent && \
echo "stratus-agent ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
Nothing after the USER stratus-agent line in the Dockerfile invokes sudo; every apt-get/pip install happens earlier, as root, before the user switch. The grant defeats the entire point of the non-root user: any process running as stratus-agent, including an LLM-driven agent issuing shell commands, can trivially become root inside the container (sudo su, sudo bash, or any command at all, since the sudoers line permits ALL=(ALL) NOPASSWD: ALL). There is no functional justification for it visible anywhere in this repository at the pinned commit, so the fix is to drop it entirely rather than narrow it to specific capabilities. The pinned Dockerfile also ships no CMD or ENTRYPOINT at all: it builds, but a container started from it with no override runs nothing beyond the base python:3.12-slim image's own default and exits immediately, which is why the Job above could not launch STRATUS even once the image existed. The real entry point is already established elsewhere on this page: pyproject.toml's [project.scripts] registers stratus/run_crew -> stratus.main:run, the same function test_agent.sh's own crewai run invokes, so that verified command is what the image should default to:
# git only: needed for `crewai install` to resolve git-based deps; no sudo.
RUN apt-get update && \
apt-get install -y git && \
rm -rf /var/lib/apt/lists/*
RUN useradd -m -u 1001 -s /bin/bash stratus-agent
# ... COPY . . and the chgrp -R 0/chmod g=u pass, unchanged from the original file ...
# Invoke the registered project script directly. CrewAI 0.95.0's `crewai run`
# wrapper catches child failures and exits 0, which is unsafe for a Job.
CMD ["uv", "run", "run_crew"]
Built and run for real in this sandbox (docker build, python:3.12-slim base, same dependency set as the pinned Dockerfile) to confirm the corrected image both builds and is genuinely unprivileged, not asserted from the Dockerfile text alone:
$ docker build -f Dockerfile.no-sudo -t stratus-nosudo-verify .
...
writing image sha256:b27a50369d54ede4dc8a51ae6a2ded75ee5c98f8f025e2064f1d9db24e007fb2 done
naming to docker.io/library/stratus-nosudo-verify done
$ docker run --rm stratus-nosudo-verify whoami
stratus-agent
$ docker run --rm stratus-nosudo-verify id
uid=1001(stratus-agent) gid=1001(stratus-agent) groups=1001(stratus-agent)
$ docker run --rm stratus-nosudo-verify which sudo
(exit 1, no output: sudo is not installed)
$ docker run --rm stratus-nosudo-verify sudo whoami
exec: "sudo": executable file not found in $PATH
$ docker run --rm stratus-nosudo-verify sh -c "touch /etc/test-write; echo exit=\$?"
touch: cannot touch '/etc/test-write': Permission denied
exit=1
The container build above validates the removal of sudo: it runs as UID 1001 with no supplementary groups, has no sudo binary, and cannot write outside the intended group-writable paths. The direct-entry correction is validated separately by the exact-source two-command regression harness above; rebuild the image after changing its CMD, and keep the explicit matching command in the Job so image and manifest cannot drift silently. No Linux capability is needed: cluster mutation comes from the Kubernetes RBAC binding, not OS privilege inside the container.
How to develop with it¶
The transaction mechanism, checkpoint, execute, commit-or-abort, undo, is small enough to validate directly without CrewAI or an LLM in the loop, and internalizing it is the prerequisite for extending STRATUS safely (a new writer agent, a different severity metric, a different confinement policy). The executed model below implements run_transaction per the paper's R1-R3 rules and visible_mu per A2's Faithful Undo, reproduces all four of Table 1's worked mitigation plans numerically5, exercises two adversarial cases (a transaction that exceeds the bounded risk window K, and a mid-transaction crash that must abort regardless of what a later action would have done), checks Lemma 3.1's invariant over 200 randomized transactions, and contrasts TNR's undo-and-retry against the paper's "naive retry w/o undo" ablation on a hand-built compounding-failure scenario:
# stratus_tnr_model.py - validated: implements the paper's Transactional
# Non-Regression (TNR) transaction semantics (Sec 3.1.1-3.1.3: A1-A3, R1-R3,
# Table 1, Lemma 3.1) and models the qualitative ablation finding that
# undo-less retry compounds severity (Sec 5, "Naive retry w/o undo"). Pure
# stdlib; models the mechanism, does not run STRATUS or CrewAI.
from __future__ import annotations
import math
import random
from typing import Sequence
CRASH = math.inf # mu(perp) = infinity, paper Sec 2
def run_transaction(mu_pre: float, deltas: Sequence[float], K: int) -> tuple[list[float], float, bool]:
"""R1 checkpoint (mu_pre) -> R2 execute a1..ak, deltas applied sequentially
-> R3 commit iff not crashed and mu_post <= mu_pre (Sec 3.1.2-3.1.3).
A3 bounds the transaction length to K (bounded risk window)."""
assert 1 <= len(deltas) <= K, f"A3 violated: transaction length {len(deltas)} exceeds K={K}"
hidden = [mu_pre]
for d in deltas:
hidden.append(hidden[-1] + d)
mu_end = hidden[-1]
committed = (mu_end < CRASH) and (mu_end <= mu_pre)
return hidden, mu_end, committed
def visible_mu(mu_pre: float, mu_end: float, committed: bool) -> float:
"""R3 + A2 (Faithful Undo): commit exposes mu_end; abort restores mu_pre
exactly, so the hidden path is invisible to other agents (Sec 3.1.2-3.1.3)."""
return mu_end if committed else mu_pre
def retry_loop(mu_baseline: float, attempts: Sequence[Sequence[float]], K: int,
use_undo: bool) -> list[float]:
"""TNR undo-and-retry (use_undo=True) re-checkpoints every attempt from the
last externally visible state; the ablation's naive retry w/o undo
(use_undo=False) chains from wherever the failed attempt's hidden path
ended, per Sec 5's ablation description."""
trace = [mu_baseline]
mu_pre = mu_baseline
for deltas in attempts:
_, mu_end, committed = run_transaction(mu_pre, deltas, K)
mu_pre = visible_mu(mu_pre, mu_end, committed) if use_undo else mu_end
trace.append(mu_pre)
return trace
# --- 1) Table 1 replication: the paper's four worked mitigation plans -------
rebalance = run_transaction(12.0, [18 - 12, 9 - 18], K=20)
assert rebalance[:2] == ([12.0, 18.0, 9.0], 9.0) and rebalance[2] is True
assert visible_mu(12.0, rebalance[1], rebalance[2]) == 9.0 # visible: 12 -> 9
upgrade = run_transaction(15.0, [22 - 15, 11 - 22], K=20)
assert upgrade[:2] == ([15.0, 22.0, 11.0], 11.0) and upgrade[2] is True
assert visible_mu(15.0, upgrade[1], upgrade[2]) == 11.0 # visible: 15 -> 11
bad_image = run_transaction(15.0, [24 - 15, 30 - 24], K=20)
assert bad_image[:2] == ([15.0, 24.0, 30.0], 30.0) and bad_image[2] is False
assert visible_mu(15.0, bad_image[1], bad_image[2]) == 15.0 # aborts: visible 15 -> 15
hotfix_ok = run_transaction(15.0, [10 - 15], K=1) # x=10 <= 15: commits
assert hotfix_ok[2] is True and visible_mu(15.0, hotfix_ok[1], True) == 10.0
hotfix_bad = run_transaction(15.0, [20 - 15], K=1) # x=20 > 15: aborts
assert hotfix_bad[2] is False and visible_mu(15.0, hotfix_bad[1], False) == 15.0
# --- 2) Adversarial: A3's bounded risk window rejects an oversized transaction,
# and a mid-transaction crash (mu -> CRASH) always aborts regardless of a
# later delta that would otherwise look like a recovery. --------------------
try:
run_transaction(15.0, [1, 1, 1], K=2)
raise SystemExit("expected AssertionError for A3 violation")
except AssertionError as e:
assert "A3 violated" in str(e)
crash_then_recover = run_transaction(15.0, [CRASH, -1000.0], K=20)
assert crash_then_recover[1] == CRASH and crash_then_recover[2] is False
assert visible_mu(15.0, crash_then_recover[1], crash_then_recover[2]) == 15.0
# --- 3) Lemma 3.1: over a long trajectory of interleaved committing and
# aborting transactions (including occasional injected crashes), the
# externally visible severity never exceeds the baseline b, regardless of
# how bad the hidden path gets inside any single transaction. -------------
rng = random.Random(7)
b = 20.0
mu_visible = b
lemma_trace = [mu_visible]
for _ in range(200):
k = rng.randint(1, 5)
deltas = [rng.uniform(-15, 25) for _ in range(k)]
if rng.random() < 0.05:
deltas[rng.randrange(k)] = CRASH # occasional injected crash attempt
_, mu_end, committed = run_transaction(mu_visible, deltas, K=20)
mu_visible = visible_mu(mu_visible, mu_end, committed)
lemma_trace.append(mu_visible)
assert all(m <= b for m in lemma_trace), "TNR invariant violated: visible severity exceeded baseline"
# --- 4) TNR undo-and-retry vs the ablation's "naive retry w/o undo" (Sec 5):
# three hallucinated fixes followed by one real fix. TNR always re-checks
# out from the last visible (undone) state, so severity never exceeds
# baseline; naive retry chains off each failed attempt's hidden end state,
# so severity compounds before the real fix ever runs. ---------------------
bad_attempts = [[8.0, 6.0], [7.0, 5.0], [9.0, 4.0], [-40.0]]
tnr_trace = retry_loop(b, bad_attempts, K=20, use_undo=True)
naive_trace = retry_loop(b, bad_attempts, K=20, use_undo=False)
assert max(tnr_trace) <= b, "TNR retry must never exceed baseline"
assert max(naive_trace) > b, "naive retry should compound past baseline in this adversarial case"
assert naive_trace[-1] > tnr_trace[-1], "naive retry ends worse off than TNR retry"
print("Table 1 replication (hidden mu-path -> commit? -> visible):")
print(f" node drain/rebalance {rebalance[0]} commit={rebalance[2]} visible=12->9")
print(f" rolling upgrade {upgrade[0]} commit={upgrade[2]} visible=15->11")
print(f" bad image attempted {bad_image[0]} commit={bad_image[2]} visible=15->15")
print(f" hotfix K=1 (x=10) {hotfix_ok[0]} commit={hotfix_ok[2]}")
print(f" hotfix K=1 (x=20) {hotfix_bad[0]} commit={hotfix_bad[2]}")
print("A3 bounded-risk-window violation raised AssertionError as expected")
print(f"mid-transaction crash always aborts: visible stays {visible_mu(15.0, crash_then_recover[1], crash_then_recover[2])}")
print(f"Lemma 3.1 over 200 transactions: max visible mu = {max(lemma_trace):.2f} <= baseline b = {b}")
print(f"retry comparison: TNR trace = {[round(x, 1) for x in tnr_trace]}")
print(f"retry comparison: naive trace = {[round(x, 1) for x in naive_trace]}")
print("all assertions passed")
Executed output:
Table 1 replication (hidden mu-path -> commit? -> visible):
node drain/rebalance [12.0, 18.0, 9.0] commit=True visible=12->9
rolling upgrade [15.0, 22.0, 11.0] commit=True visible=15->11
bad image attempted [15.0, 24.0, 30.0] commit=False visible=15->15
hotfix K=1 (x=10) [15.0, 10.0] commit=True
hotfix K=1 (x=20) [15.0, 20.0] commit=False
A3 bounded-risk-window violation raised AssertionError as expected
mid-transaction crash always aborts: visible stays 15.0
Lemma 3.1 over 200 transactions: max visible mu = 20.00 <= baseline b = 20.0
retry comparison: TNR trace = [20.0, 20.0, 20.0, 20.0, -20.0]
retry comparison: naive trace = [20.0, 34.0, 46.0, 59.0, 19.0]
all assertions passed
The retry comparison is the point worth sitting with: after three bad attempts, TNR's visible trace never leaves the baseline (20.0 throughout), because each abort is invisible; the naive trace climbs to 59.0, a system nearly three times as sick as when detection started, before the fourth, genuinely good, attempt only claws it back to 19.0, still worse than TNR's -20.0 (fully resolved) from the same four attempts. This is the executed shape of the paper's own finding on real problems: undo-less retry from AIOpsLab's ablation reaches only 23.1% success against TNR's 69.2%.12
To extend STRATUS: a new writer agent must acquire the A-Lock like αM/αU (no exceptions, or A1 is void); a new severity metric only needs to preserve μ(⊥) = ∞ and monotonicity under the intended alert/violation/capacity signals for Lemma 3.1's proof to still apply; and a new confinement rule set should be additive to Table 5, not a replacement, since the paper's own limitation is incompleteness, not overreach.7
Running it in production¶
The paper does not describe a customer-facing production deployment; the evaluation is against two benchmark harnesses' emulated clouds (DeathStarBench SocialNetwork/HotelReservation via AIOpsLab, and ITBench's scenario set), 15-20 machine hours per suite.10 What it does specify, and what a real rollout should copy, is the guardrail stack that makes autonomous write access defensible: role-based confinement (readers cannot mutate, ever), a dry-run preview (kubectl ... --dry-run=server) before every mutating command that also happens to be how interactive commands are rejected today, per "The real confinement layer, and a genuine defect in it" above, not a standalone Table-5-shaped linter, and the combined-oracle termination check (alert clearance, user-facing error rate, and object-level health, all three, not any one).789 STRATUS's own design explicitly supports a lower-autonomy fallback: "hybrid operations where writer agents can request human approval before execution, with interceptable action stacks and tool interfaces", the natural hook for wiring a risk-tiered approval gate in front of αM for actions your policy classifies above auto/validate.7 Before granting write access to anything real: fix or replace KubectlLinter (it cannot currently be instantiated, see above), and bind the process to a dedicated ServiceAccount/Role/RoleBinding scoped to the target namespace rather than running with whatever KUBECONFIG happens to be on the host, since the shipped test_agent.sh flow does the latter by default.
For container deployment, the shipped Dockerfile (python:3.12-slim, non-root UID 1001, OpenShift arbitrary-UID-compatible via chgrp -R 0/chmod -R g=u, corrected above to drop its unused sudo grant and to set a real CMD) is a real starting point. The repository itself still ships no Kubernetes manifest, but this page does: the RBAC-scoped ServiceAccount/Role/RoleBinding/ClusterRole in "A real, least-privilege RBAC manifest" above and the Job in "Workload shape" above are both server-validated and were actually applied against a live cluster, not left as an exercise for the reader.
Sizing and cost: at GPT-4o, budget roughly 800-1,700 seconds and $0.88-$6.11 per mitigation problem depending on benchmark complexity, with retries (up to 9) accounting for most of both. Whether GPT-4o-mini saves money is benchmark-dependent, not a general property of the cheaper model: on AIOpsLab it cuts cost by roughly 24x ($0.036 vs $0.877) because its step count only grows about 2.7x (125.7 vs 46.3 steps) to compensate for weaker per-step performance; on ITBench it costs 53% more ($9.38 vs $6.11) because its step count blows up about 4x (468.9 vs 115.7 steps), outrunning the per-token price cut. Success rate falls by more than half on both benchmarks (69.2% to 23.1% on AIOpsLab, 50.0% to 19.4% on ITBench). So model choice is a genuine success/cost trade, and on the harder benchmark it is not even a cost saving.11 The current concurrency model is one STRATUS instance per managed system (one namespace, single writer at a time); running many instances against overlapping infrastructure needs the Sec 3.3 resource-locking extension the paper describes but does not implement, since TNR's guarantee as published does not extend to concurrent writers without it.6
Failure modes¶
| Pitfall | Cause | Fix |
|---|---|---|
| Undo is assumed faithful but is not always achievable | Complex or externally-visible side effects (application state, third-party API calls) do not reduce to a simple state-object rollback; the paper names this a practical limitation, not a solved problem | Scope autonomous write access to reconciliation-based resources (Kubernetes objects and similar) where rollback is tractable; require human approval for anything with side effects outside the platform's own state model |
| A benchmark "win" can be a benchmark artifact | In 8 of 18 ITBench mitigation problems, STRATUS succeeds by restarting failing pods, which clears the alert because the fault injector cannot re-target a restarted pod, not because the underlying fault (a persistent misconfiguration or bad image) was fixed | Do not report ITBench mitigation success as evidence of fixing persistent faults; re-verify against faults that are known to survive a pod restart before trusting the number |
| Weaker models blow the retry budget without matching gains | STRATUS (GPT-4o-mini) took 3,557.9s and 125.7 steps on AIOpsLab mitigation for 23.1% success, versus GPT-4o's 811.9s/46.3 steps for 69.2%; more retries did not close the capability gap | Benchmark the exact model you plan to deploy, not just the flagship the paper reports; set a step/time ceiling per problem class informed by your own model's plateau, not the paper's K=20 default |
| Single-writer lock is a scaling ceiling | Writer Exclusivity (A1) is a strict readers-writer lock; the paper's current deployment is one STRATUS instance per namespace with no concurrency control implemented | Shard by namespace/system as the paper does, or implement the Sec 3.3 resource-set admission controller (Admit(Pi) <=> R(Pi) ∩ Rlocked = ∅) before running multiple concurrent mitigations against shared resources |
| RCA scores understate diagnosis quality | AIOpsLab's RCA oracle grades against non-mutually-exclusive category labels; the paper documents a case where STRATUS's answer is arguably more precise than the oracle's expected label but is scored wrong regardless | Treat AIOpsLab RCA percentages as a noisy lower bound, not a diagnosis-accuracy ground truth; spot-check disagreements before concluding a diagnosis is actually wrong |
| Confinement is necessary, not sufficient | The paper states plainly that its rule-based confinement (Table 5) "may not be perfect to capture all destructive actions"; at commit 4fc9a5b66be, KubectlLinter() cannot even be constructed (AttributeError: ResourceType has no attribute REPLICASET, confirmed by direct instantiation) |
Layer independent guardrails, a policy engine or risk-tiered approval, in front of the mitigation agent; do not rely on KubectlLinter until the REPLICASET enum gap is fixed upstream or in your fork, and confirm confinement is coming from the dry-run path in kubectl.py, not the linter, at whatever commit you deploy |
| Cluster-admin by default | test_agent.sh exports KUBECONFIG=$HOME/.kube/config with no RBAC manifest shipped, so a fresh kind bring-up runs STRATUS with cluster-admin credentials |
Apply the real, server-validated rbac.yaml/job.yaml manifests above and bind serviceAccountName: stratus-agent before pointing it at anything beyond a disposable test cluster; verify the boundary with kubectl auth can-i --as, not by reading the YAML |
References¶
- Chen, Pan, Clark, Su, Zheutlin, Bhavya, Arora, Deng, Jha, Xu, STRATUS: A Multi-agent System for Autonomous Reliability Engineering of Modern Clouds (NeurIPS 2025, arXiv 2506.02009): https://arxiv.org/abs/2506.02009
- STRATUS repository, pinned at commit
4fc9a5b66bebbc5ddf2154c06e97f05db9385183(2025-10-23), Apache 2.0: https://github.com/xlab-uiuc/stratus - AIOpsLab repository, the git submodule STRATUS's AIOpsLab adapter runs against: https://github.com/xlab-uiuc/AIOpsLab/
- Chen, Shetty, Somashekar, Ma, Simmhan, Mace, Bansal, Wang, Rajmohan, AIOpsLab: A Holistic Framework to Evaluate AI Agents for Enabling Autonomous Clouds (MLSys 2025, arXiv 2501.06706): https://arxiv.org/abs/2501.06706
- Jha, Arora, Watanabe, Yanagawa, Chen, Clark, Bhavya, et al., ITBench: Evaluating AI Agents across Diverse Real-World IT Automation Tasks (ICML 2025, arXiv 2502.05352): https://arxiv.org/abs/2502.05352
- CrewAI (multi-agent framework STRATUS is implemented on): https://www.crewai.com/
- Alpern and Schneider, Recognizing Safety and Liveness (Distributed Computing, 1987), the formal safety-property class TNR instantiates: https://doi.org/10.1007/BF01782772
Related: AIOpsLab · Agentic incident management: OpsAgent · Agentic AIOps · Risk-tiered human approval gates · Multi-agent collaboration · Agent policy engine · Agent intent verification · Governing self-modifying agents · Evaluation integrity and anti-gaming · Operational runbooks
-
Sec 1 (Introduction) and the paper's contributions list: STRATUS differs from prior human-assist AIOps tools (data collection, summarization, root-cause prediction) by targeting autonomous mitigation of live systems; multi-agent design chosen for role specialization, customizability/extensibility, and easier agent-safety reasoning; deterministic state-machine control flow with LLM-driven data flow per agent; artifacts at https://github.com/xlab-uiuc/stratus. ↩
-
Sec 2 (Background and Problem Definition): environment
Ewith statesS, crash state⊥; severityμ(s) = w1|A| + w2|V| + w3|L|(alerts, SLA violations, unhealthy-node capacity loss),wi > 0,μ(⊥) = ∞; four SRE tasks (detection, localization, RCA, mitigation); RCA explicitly stated as not on the mitigation critical path and typically done offline. ↩ -
Sec 3 (STRATUS Design), first two paragraphs: agent set
A = {αD, αG, αM, αU}with responsibilities (αD initial error state; αG localization+RCA; αM plan/decompose/execute mitigation; αU undo sequence on abort); action spaceAread/Awrite/Aundovia ACI overkubectl; αD, αG restricted toAread; state-transition relationR : S x Awrite -> S ∪ {⊥}. ↩ -
Sec 3.1 (Safety Specification): assumptions A1 Writer Exclusivity (A-Lock, readers-writer lock over αM/αU), A2 Faithful Undo (
U(spost) = spreexactly), A3 Bounded Risk Window (transaction lengthk,1 <= k <= K); transaction rules R1 Checkpoint, R2 Execute, R3 Commit/Abort (commit iff spost != ⊥ and μ(spost) <= μ(spre), else abort via αU); Lemma 3.1 (induction proof that every externally visible state satisfiesμ(s) <= b, the severity at initial detection); TNR framed as an Alpern-Schneider safety property (Sec 3.2). ↩↩ -
Table 1 (the four worked mitigation plans): Node drain/rebalance (cordon, evict, scale), hidden mu-path 12 -> 18 -> 9, commit, visible 12 -> 9; Rolling upgrade (scale 0, patch, scale 3), hidden 15 -> 22 -> 11, commit, visible 15 -> 11; Bad image attempted (scale 0, patch(bad), scale 3), hidden 15 -> 24 -> 30, abort, visible 15 -> 15; Single hot-fix (K=1, apply hotfix), hidden 15 -> x, commit iff x <= 15 else abort, visible <= 15 either way. ↩
-
Sec 3.3 (Extension): current deployment is one STRATUS instance per cloud system (separate namespace), no concurrency control between writer agents; described (not implemented) extension is a Coordination Controller / distributed lock manager over per-plan resource sets
R(Pi), admission ruleAdmit(Pi) <=> R(Pi) ∩ Rlocked = ∅, releasing resources on transaction commit/abort. ↩↩ -
Sec 4 (Implementation) and 4.1 (Realizing TNR): built on the CrewAI multi-agent framework, state-machine control flow (Fig 2), off-the-shelf LLMs (GPT, Llama) with no fine-tuning; Writer Exclusivity via role-based sandboxing (readers cannot mutate) plus state-machine mutual exclusion for writer agents, with
kubectl --dry-runused where applicable; Faithful Undo via a stack-based rollback mechanism (Fig 3: push/pop of(command, resource_type, state)tuples, LIFO order), destructive irrecoverable actions rejected or rewritten to a recoverable form (file delete -> move to backup volume); explicit limitation that "our rule-based confinement may not be perfect to capture all destructive actions"; hybrid-operations note that writer agents can request human approval before execution with interceptable action stacks and tool interfaces. ↩↩↩↩↩↩↩ -
Appendix A (Confinement Rules) and Table 5: read-only agents restricted to non-mutating commands (
kubectl get/describe/logs,ls,cat, TelemetryCollection); writer agents restricted to sequential (serialized) execution; blocked for all agents: namespace deletion,kubectl debug/edit, stdin redirection (-f -), interactive flags (--stdin,--tty,exec -it), shell pipes, compound commands (&&,||,;), command substitution, flow control, shell function definitions; static linting validates syntax before execution; in evaluation, 10.5% of STRATUS (GPT-4o)'s commands and 9.7% of STRATUS (Llama 3.3)'s would have caused system hangs absent confinement. ↩↩ -
Sec 4.2 (Other Notable Implementations) and Appendix D-E: observability tools (
GetLogsTool,GetTracesTool/ReadTracesTool,GetBootstrapAnalysisTool/ReadBootstrapAnalysisTool,OracleTools) and command-line tools (NL2Kubectl,LintingTool,RollbackTool,NotificationTool); direct raw-metric consumption disabled, information surfaced via pre-processed logs/summaries instead; trace-based bootstrapping builds a call graph to seed the diagnosis agent's initial fault-localization hypothesis; termination combines three "weak" oracles (alert clearance, user-request success/no 4xx-5xx, system health) into one pass/fail gate, illustrated in Appendix E'swrong_bin_usage-mitigation-1example (first oracle check: 115/117 failed responses despite healthy pods; second check after patch: 0/120 failed). ↩↩↩ -
Sec 5 (Evaluation), opening paragraphs: benchmarks AIOpsLab and ITBench, each a live emulated-cloud arena with per-problem oracles; baselines AOL-agent and ITB-agent (each suite's reference agent), plus ReAct and Flash from AIOpsLab; models GPT-4o (
gpt-4o-2024-08-06), GPT-4o-mini (gpt-4o-mini-2024-07-18), Llama 3.3 (llama-3-3-70b-instruct), temperature 0; metrics success rate, time, steps, USD cost; each benchmark took STRATUS 15-20 machine hours to complete. ↩↩↩ -
Table 2 (mitigation results): AIOpsLab, 13 problems, ReAct(4o) 23.1%/46.0s/23.0 steps/$0.112, Flash(4o) 38.5%/154.0s/23.1/$0.150, AOL-agent(4o) 46.2%/223.3s/21.7/$0.206, AOL-agent(mini) 7.7%/58.9s/22.7/$0.003, AOL-agent(llama) 15.4%/98.2s/13.0/$0.037, STRATUS(4o) 69.2%/811.9s/46.3/$0.877, STRATUS(mini) 23.1%/3557.9s/125.7/$0.036, STRATUS(llama) 23.1%/1486.9s/71.8/$0.360; ITBench, 18 problems, ITB-agent(4o) 9.2%/251.7s, ITB-agent(llama) 5.7%/440.8s, STRATUS(4o) 50.0%/1720.8s/115.7/$6.11, STRATUS(mini) 19.4%/3874.9s/468.9/$9.38, STRATUS(llama) 28.0%/2566.6s/160.3/$0.76; text states 1.5x (AIOpsLab) and 5.4x (ITBench) improvement over second-best. ↩↩↩
-
Table 3 (AIOpsLab ablation, GPT-4o): STRATUS 69.2%/811.9s/$0.877; No retry 15.4%/72.6s/$0.163; Naive retry w/o undo 23.1%/1221.5s/$0.929. Table 6 (ITBench ablation, GPT-4o): STRATUS 50%/1720.8s/$6.11; No Retry 11.1%/267.2s/$0.72; Naive retry w/o undo 50%/1711.9s/$5.13 (same task-level success because 8/18 problems are solved by pod-restart alone, independent of undo). Fig 5: retry-count PDF, limit of 9 retries, 80+% of problems retry at least once, 30+% retry 5 or more times. Sec 5 prose: in the 8 ITBench problems, restarting failing pods one-by-one clears alerts because the fault injector cannot recognize the original pod; among issued mitigation actions in those 8, ~15.2% are pod deletions auto-recreated by the Deployment controller, ~44.2% are hallucinated/wrong values (e.g. a nonexistent image name) fixed on the next retry via the reflection mechanism. ↩↩↩↩
-
Table 4 (AIOpsLab, other SRE tasks, 86 problems total: 32 detection, 28 localization, 26 RCA): STRATUS(4o) Detection 90.6%/48.4s/$0.118, Localization 51.2%/65.3s/$0.126, RCA 34.6%/39.6s/$0.068; STRATUS(mini) 78.1%/25.0%/30.8%; STRATUS(llama) 93.8%/36.3%/26.9%; baselines ReAct(4o) 87.5%/26.8%/23.1%, Flash(4o) 59.4%/39.3%/26.9%, AOL-agent(4o) 62.5%/46.9%/38.5%. Localization scored Pass@3. RCA-label-ambiguity example: a k8s target-port misconfiguration's oracle expects layer "Virtualization" and fault type "Misconfiguration"; STRATUS answers layer "Application" and fault type "Network/Storage Issue", scored incorrect despite the paper arguing STRATUS's answer is more precise (no hypervisor-style virtualization layer exists in AIOpsLab). ↩
-
Appendix B (Bounded Risk Window) and Fig 6: step-limit sweep over {3, 5, 10, 15, 20, 30} on AIOpsLab across all four task types; STRATUS (GPT-4o) improves from 0% to 53.53% success moving from a 3-step to a 10-step limit; performance plateaus past 15 steps across evaluated agents; K=20 set as the default from this analysis. ↩