AOI: trainable multi-agent cloud diagnosis (Autonomous Operations Intelligence)¶
Scope: AOI (Autonomous Operations Intelligence, arXiv 2603.03378), a trainable multi-agent SRE framework built from a read-write separated Observer/Probe/Executor/Compressor runtime, Observer-level GRPO training, and a Failure Trajectory Closed-Loop Evolver, evaluated on AIOpsLab's 86-task benchmark. Covers the permission architecture, the GRPO advantage and reward formulation, the Evolver's repair-versus-augmentation split, and the paper's own results tables and ablations. This is a distinct project from the other same-acronym paper in this KB, AOI: AI-Oriented Operations (arXiv 2512.13956, a different research group); the two papers share only a three-letter name, not authors, architecture, or results, and should never be conflated. For the broader field context see agentic AIOps and its production-deployed worked example OpsAgent; the training algorithm itself is GRPO; the read-write separation pattern generalizes the isolation principles in agent sandboxing and isolation.
All benchmark numbers on this page (Tables 2-4, Appendices B-E) are the paper's own AIOpsLab runs; no AIOpsLab cluster or Qwen3-14B training run was reproduced here. Three Python blocks are executed and asserted: one models the Table 1 agent/store permission matrix, Algorithm 1's main loop, and the Executor's evidence-and-available-actions gate (not a whitelist; see "The real Executor gate" below for why); a second reproduces AIOpsLab's real two-layer command gate (method-registry check plus the
exec_shelldenylist) verbatim from the submodule source, including the adversarial case the denylist does not catch; the third models the Eq. 1 GRPO advantage, the Eq. 3 six-dimension step reward with its hard-penalty rule, and the best@k/avg@k metrics applied to a population constructed to honor Table 9's stability-bucket counts (the paper publishes only bucket totals, not per-round pass/fail identity per task, so the reconstructed best@k curve is a metric-definition check, not a reproduction of Figure 7). Code status: the PDF's own "Code and Data Availability" section says the review repository was anonymized (anonymous.4open.science/r/aoi-C8C7, returns HTTP 401 as of 2026-07-16); the de-anonymized project page named in the arXiv front matter,https://github.com/OpenEdgeHQ/aoi, is live and public, cloned at commit17c8f55a030e8850b93c9d16074c2976b6384619(2026-02-11; the repository ships noLICENSEfile, check with the authors before redistributing), and its README describes the same Observer/Probe/Executor/Compressor architecture, so that repository is the one cited below.Independently verified here beyond the paper: every file quoted in "How to bring up the released repository" (
.env.example,config/agent_config.py,grpo/observer/grpo_config.py,grpo/evolver/grpo_config.py,grpo/observer/reward_model.py,agents/executor_agent.py,environment/aiopslab_server.py,environment/aiopslab_server2.py,main_aiopslab.py,README.md) is the real source, read directly from the clone.ObserverGRPOConfig()and itsgroup_size < 2validation were constructed and exercised directly in this sandbox (no GPU needed, the class is a plain dataclass); the real, effective batch size (32) and epoch count (5) it computes were read from the executed object, not the docstring. A repository-widegrepfor "whitelist" (and its aliases) across every.py/.yamlfile, including theAIOpsLabsubmodule pinned ata56bb5db5d28348dba2ea66ae7693c0b4ee6e6ac, found zero matches: the paper's Table 5 claim of a "47-pattern Executor whitelist" does not correspond to any command-whitelist artifact in this repository at this commit; the real constraint on Executor actions isavailable_actions: Dict[str, str], AIOpsLab's own per-problem API registry, a different mechanism, documented below rather than asserted to still exist, and traced down to the actualgetattr-based method gate and the 5-entryexec_shelldenylist that implement it. This session additionally re-cloned the repository fresh, read the FastAPI serving app (environment/aiopslab_server.py) endpoint by endpoint, and validated a least-privilege RBAC/Deployment/Service manifest set against a real, disposablekindcluster created for this check (kind create cluster --name aoi-verify, later deleted), including a realkubectl rollout undorollback and a Service-to-pod endpoint wiring proof. No AIOpsLab cluster, GPU, or LLM API key was available in this sandbox, so no training run, live inference serving, or held-out evaluation was executed end to end here; the checkpoint/rollbackHTTP endpoint's code path was read from source but not called live, since that requires an activeinit_problemsession this sandbox cannot start.2026-07-17 follow-up, closing four defects a later adversarial audit found in the paragraph above. The audit observed that
aiopslab_server2.pysetsKUBECONFIGfrom a Kind cluster's admin credentials at module-import time, unconditionally, so a Pod running this server never actually uses its own scoped ServiceAccount; that the Deployment below had only ever reachedImagePullBackOff(no image was built); that no rollback had run against a genuinely live Deployment; and that theexec_shellgate could still read in-process secrets or run commands outside its 5-entry denylist. All four were re-driven end to end in this follow-up, against a freshly built image and a new disposable cluster (kind create cluster --name aoi-kb-fix, later deleted). First, both places that unconditionally shelled out tokind(the module-levelensure_kind_cluster_and_kubeconfig()call andEnvironmentServer.startup()'screate_kind_cluster()call) were patched, in a real clone at the pinned commit, to detect in-cluster execution the waykubernetes.config.load_incluster_config()does and, when in-cluster, build a kubeconfig from the Pod's own mounted ServiceAccount token instead of ever touching Kind or an admin credential; see "The in-cluster credential guard" below for the patch and "A real, least-privilege RBAC manifest" for aSelfSubjectAccessReviewexecuted from inside the running Pod, over that exact generated credential, that reproduces the Role's allow/deny boundary exactly. Second, that same patched source was built into a real ~2 GB image (docker build,python:3.11-slim, pinned requirements installed with--no-depsto route around an unresolvableaiohttpconflict betweenaworld==0.2.6andauth0-python), loaded into the new cluster withkind load docker-image, and the Deployment reached genuine1/1 Runningwith a passing/healthreadiness probe, notImagePullBackOff; see "Getting the Deployment to Running" below. Third, a real bad rollout (kubectl set image ... aoi-executor:local-bad, producing a liveErrImageNeverPull) was undone withkubectl rollout undoagainst that genuinely-running Deployment, restoring1/1 Runningand a passing/healthcheck through the Service's ClusterIP; a further, honest attempt to exercise the/checkpoint//rollbackHTTP pair by actually calling/init_problemsurfaced a real, different, and more specific boundary than "not exercised": it correctly hit a403 Forbiddenfrom the scoped ServiceAccount trying to list pods in theopenebsnamespace (proving the least-privilege Role holds under real load), which then blocked the single-worker event loop long enough to fail its liveness probe;/checkpointand/rollbackthemselves were still never reached. See "A real rollback command" below for both. Fourth, theexec_shellgate's exposure (unrestricted secret reads viaenv/printenv/cat /proc/self/environ, command injection via;/&&, and destructive commands outside the 5-entry denylist) was closed with a real, executed, adversarial argv-level allowlist that rejects every one of those bypasses while still accepting the Executor's three legitimate actions; see "A real allowlist-based tightening of the exec_shell gate" below.
What it is¶
AOI targets three obstacles the authors give for why LLM-agent SRE has not reached enterprise production: proprietary operational data cannot leave the estate, so only small locally-deployed models (under 100B parameters) are viable; permission-governed environments make unsafe action execution a hard blocker, not a tuning problem; and closed, static systems cannot improve from their own failures.1 It answers with three integrated pieces: a trainable diagnostic system that applies Group Relative Policy Optimization (GRPO) to distill expert-level diagnostic behavior into a locally-deployed open-weight model; a read-write separated execution architecture that keeps diagnosis and remediation on strictly different privilege rails; and a Failure Trajectory Closed-Loop Evolver that mines the system's own unsuccessful trajectories and turns them into corrective training signal and inference-time guidance.1
The runtime (Section 3) is four specialized agents coordinating over three memory stores, not a single prompted LLM. Observer plans, tracks hypotheses, and decides the next action (Probe, Execute, or Submit), but never touches the environment directly. Probe issues read-only commands (kubectl get, describe, logs) with up to Kmax=5 exploration rounds per iteration. Executor issues state-changing commands behind a whitelist filter, with two-stage error recovery and a "look before you leap" option to invoke a single Probe round before acting. Compressor deduplicates and semantically compresses raw tool output into the Observer's sole information source, and is stateless: every iteration is processed independently so errors cannot silently accumulate across the trajectory.2 Table 1's access-control matrix over three stores, Mraw (raw environment output), Mtask (task queue and hypothesis state), and Mcomp (compressed context), makes the separation structural rather than a prompt convention: Observer can only read Mcomp; Probe and Executor can write Mraw but not read it back; only the Compressor moves information from Mraw into Mcomp.2
Long-horizon coherence comes from dual-timescale memory: long-term memory H is a running list of per-iteration summaries S_i, and short-term memory is the full compressed context C_{n-1} from the immediately preceding iteration; at iteration n the Observer conditions its decision on (H_{n-2}, C_{n-1}).2 Each iteration is a fixed four-stage pipeline: Decision (Observer emits an action and instruction), Interaction (the routed agent executes in the environment and writes Mraw), Compression (the Compressor produces C_n within a fixed token budget), and Caching (the platform stores C_n and appends the Observer's summary to H).2
GRPO trains two different things at two different granularities. Observer GRPO (Section 3.3) optimizes single-step decisions: for a context x, a group of G candidate actions {y_i} are sampled, an LLM judge scores each on six dimensions, and the group-normalized advantage A_i = (R(x,y_i) - mu_G) / (sigma_G + eps) drives the policy gradient.3 Evolver GRPO (Section 4) optimizes whole-trajectory generation: given a seed trajectory tau, it samples G candidate corrected command sequences and scores them on validity, completeness, correctness, and effectiveness, using the same group-relative advantage form.4 Training data for Observer GRPO comes only from Evolver-repaired or Purifier-cleaned successful trajectories; the Purifier strips retries and dead-end exploration from a successful run down to the minimal command sequence that reached the correct diagnosis.1
Why use it¶
- Read-write separation alone is worth more than the base model. With an identical GPT-4o-mini base and no task-specific training, AOI reaches 58.1% overall on the full 86-task benchmark against the reference ReAct-style AOL-agent's 14.7%, a roughly 4x improvement the paper attributes to Observer being free to explore diagnostically without risking a state mutation.6 On the Qwen3-14B (best@5) rows, the same architectural change takes Detection from STRATUS's 75.0% to 100%, RCA from 7.7% to 30.8% (a 4x / +300% relative gain -- note the paper's own "+150% over STRATUS" RCA claim in Section 5.2 arithmetically matches the GPT-4o-mini row's RCA change instead, 15.4% -> 38.5%, not this Qwen3-14B row), and Mitigation from 15.4% to 46.2% (3x), because Executor-level safety gates prevent the cascading failures STRATUS suffers when it mutates state before diagnosis is complete.6
- A 14B open-weight model with this architecture beats a frontier model with a simpler one. Qwen3-14B + AOI reaches 66.3% best@5 overall (solving 57/86 tasks) against Claude Sonnet 4.5 + AOL-agent's 57.0% (49/86), a 9.3-point gap. (The abstract's headline "24.4 percentage points" figure is a different comparison -- it is the gap between AOI's untrained runtime, 66.3% best@5, and the prior state-of-the-art STRATUS's Qwen3-14B best@5 Overall score, 41.9%, not a comparison against Claude Sonnet 4.5.)6 Sonnet still wins outright on Mitigation (76.9% vs AOI's 46.2%) and RCA is where it is weakest (15.4%, tied with GPT-4o-mini+STRATUS), which the paper reads as frontier models being strong at remediation pattern-matching but no better than a well-scaffolded 14B model at multi-step causal reasoning.6
- Observer GRPO training generalizes to fault types it never saw. Trained on only 23 tasks across 11 fault types, the 14B Observer lifts avg@1 from 33.7% (untrained AOI) to 42.9% on 63 held-out tasks spanning 15 unseen fault types, surpassing Claude Sonnet 4.5's 41.3% on the identical held-out set without any multi-run sampling.7 The gain concentrates in Detection (54.5% -> 90.9% vs Sonnet, +36.4pp) and RCA (8.3% -> 16.7%, +8.4pp), the two task types the paper argues reward systematic diagnostic search over pattern recognition; Mitigation stays flat at 14.3% because Observer optimization cannot directly improve commands the Executor generates.7
- The Evolver converts failure into a measurable reliability gain, not just a headline accuracy bump. On the 37 tasks Claude Sonnet 4.5 originally failed, adding Evolver-generated corrected plans as prompts raises end-to-end avg@5 by 4.8 points (24.9% -> 29.7%) and shrinks the best@5-avg@5 gap from 29.2pp to 18.9pp, a 35% reduction the paper reads as reproducibility rather than luck: Detection's own gap alone halves from 50pp (100% best@5, 50.0% avg@5) to 26pp (90% best@5, 64.0% avg@5).8 Repair quality itself improves under GRPO training: an LLM-judge score of the Evolver's repaired plans rises from a mean of 7.18 to 8.27 (out of 10) while its standard deviation drops from 0.97 to 0.49, which the paper reads as the trained Evolver learning structural correction patterns rather than memorizing command strings.8
When to use it (and when not)¶
- Use the pattern where an SRE agent needs write access to a real environment and proprietary telemetry cannot leave the estate; the whole design exists to make a sub-100B locally-deployed model safe and competent enough to replace a closed frontier API for this constraint.1
- Use Observer GRPO when you have even a small set of successful trajectories (the paper trained on 23 tasks) and need the gains to transfer to fault types outside that set; the held-out evaluation is deliberately a strict fault-type split with zero overlap between train and test types.5
- Do not expect uniform gains from GRPO training. It is not free lunch: the same training that lifts Detection by 25.5 points (untrained AOI 65.5% -> trained 90.9%, avg@1 per the abstract's and Section 5.3's own framing of these Table 3 numbers -- though Figure 10's caption in the same paper labels this identical delta "best@5," a source inconsistency flagged here rather than silently resolved) drops Localization by 4.5 points (22.7% -> 18.2%), because the trained Observer learns to explore roughly 9 more steps per task on average, which helps find an anomaly but hurts pinpointing the faulty component among several similar candidates; both of the paper's two GRPO-degraded tasks are Localization instances that flip from 4/5 to 0/5 successes.9 Net effect across the 63 held-out tasks is +9 (11 improved, 2 degraded), positive but not monotonic, so re-run the paper's per-task-type breakdown on your own fault mix before trusting an aggregate delta.9
- Do not treat this as a validated production system yet. Unlike this KB's OpsAgent page, which documents 53 days of live Lenovo deployment, AOI's own Limitations section states production deployment is future work ("we plan to deploy AOI in production SRE environments to validate the use and productization potential"); every number on this page is an AIOpsLab benchmark run.10
- Expect a hard capability floor. 29 of 86 tasks (33.7%) never succeed across five sampling rounds, GPT-4o-mini, or any GRPO variant tested; the paper characterizes these as systematic capability gaps (MongoDB auth recovery needing Helm-specific knowledge, multi-service localization needing causal reasoning the architecture does not yet provide) rather than noise, and recommends routing them to human escalation rather than more retries.9
Architecture¶
flowchart TB
SRE["SRE troubleshooting workflow<br/>seed trajectory"] --> JUDGE{"Judge: classify outcome"}
JUDGE -->|"success, 49 seeds"| PURIFIER["Purifier: strip retries and<br/>dead-ends to minimal sequence"]
JUDGE -->|"failed, 37 seeds"| EVOLVER["Trajectory Evolver:<br/>GRPO-sampled repair, G=4"]
PURIFIER --> TRAINDATA["Observer GRPO training set<br/>23 tasks, 11 fault types"]
TRAINDATA --> GRPOTRAIN["Observer GRPO:<br/>group-normalized advantage,<br/>6-dim step reward"]
EVOLVER --> CORRECTED["Corrected diagnostic plan<br/>structured prompt"]
GRPOTRAIN --> OBSERVER
CORRECTED -.->|"guidance at inference"| OBSERVER
subgraph RUNTIME["AOI Multi-Agent Runtime"]
OBSERVER["Observer: decide Probe, Execute<br/>or Submit; reads Mcomp only"] -->|"Probe"| PROBE["Probe: read-only kubectl<br/>get/describe/logs, up to 5 rounds"]
OBSERVER -->|"Execute"| EXECUTOR["Executor: whitelisted<br/>state-changing commands"]
PROBE --> MRAW[("Mraw: raw environment output")]
EXECUTOR --> MRAW
MRAW --> COMPRESSOR["Compressor: stateless dedup<br/>plus LLM semantic compression"]
COMPRESSOR --> MCOMP[("Mcomp: compressed context")]
MCOMP --> OBSERVER
end
OBSERVER -->|"Submit"| RESULT["Root-cause / mitigation submission"]
RESULT --> JUDGEThe left half is Figure 1's Closed-Loop Evolution Pipeline: a Judge classifies each completed SRE workflow, successful ones get purified into Observer GRPO training data, failed ones get repaired by the Evolver into structured guidance the Observer receives as a prompt at inference time. The right half is Figure 2's runtime: Algorithm 1's main loop, in which Observer never sees Mraw directly and every diagnostic or remediation action is routed through the Compressor before Observer sees the result.2 Budget exhaustion is explicit in the algorithm, not a hang: after N=15 iterations (Table 5) without a Submit, the loop returns a timeout submission.5
How to use it¶
The reference configuration trains and serves a single open-weight model: Qwen3-14B as the base, LoRA fine-tuning at rank 64 / alpha 128 / learning rate 1e-5, GRPO group size G=4, batch size 16, 3 epochs, on 2xA100 GPUs with vLLM for inference; the Evolver's reward model is Claude Opus 4.5 used only as an offline judge, never in the serving path.5 Table 5's runtime hyperparameters are the starting point for a new deployment: 15 max iterations, up to 5 Probe rounds per iteration, a 4096-token context budget per iteration, 10-summary long-term memory capacity, and a "47-pattern Executor whitelist" the paper describes but the released repository, at the pinned commit below, does not ship as any findable artifact; see "The real Executor gate" below for what actually constrains it.5 The exact bring-up commands, configuration schemas, and training entry points are in "How to bring up the released repository" below.
How to bring up the released repository¶
Clone, submodule, and install¶
git clone --recurse-submodules https://github.com/OpenEdgeHQ/aoi.git
cd aoi
git checkout 17c8f55a030e8850b93c9d16074c2976b6384619 # 2026-02-11
pip install -r requirements.txt # 240+ pinned packages: crewai==0.165.1, aworld==0.2.6,
# kubernetes==32.0.1, litellm==1.74.9, chromadb==0.5.23, ...
cd AIOpsLab && pip install -e . && cd ..
requirements.txt pins the agent framework to aworld==0.2.6 (imported directly in agents/executor_agent.py as from aworld.agents.llm_agent import Agent), not the CrewAI framework STRATUS uses on the same AIOpsLab benchmark; crewai/crewai-tools are also pinned but are not the runtime this repository's own agent classes import from.
The real configuration surface¶
.env.example is the actual environment-variable contract: OPENROUTER_API_KEY/OPENROUTER_BASE_URL/OPENROUTER_MODEL (the primary path), OPENAI_API_KEY as an alternative, commented-out DEEPSEEK_API_KEY/GROQ_API_KEY/DASHSCOPE_API_KEY backends, MODEL (defaults to gpt-4o-mini-2024-07-18 in the template, anthropic/claude-sonnet-4.5 per the README's own worked example), SUPERVISOR_MODEL, and USE_WANDB. Two real, small dataclasses (config/agent_config.py) are the inference-time config every run actually constructs, quoted directly:
@dataclass
class RollbackConfig:
validate_rollback: bool = True
retry_wait_time: int = 5
clear_replicaset: bool = True
clear_rs_wait_time: int = 10
output_dir: str = "./rollback_output"
namespace: Optional[str] = None
@dataclass
class AgentSystemConfig:
api_key: str
model_name: str = "gpt-4o-mini"
base_url: str = "https://api.openai.com/v1"
temperature: float = 0.1
max_steps: int = 20 # note: paper's Table 5 states max_iterations=15
memory_provider: str = "aworld"
rollback_config: Optional[RollbackConfig] = None
AgentSystemConfig.max_steps defaults to 20 in the shipped code, not the 15 Table 5 reports as the paper's own default; if you need the paper's exact budget, pass max_steps=15 explicitly rather than trusting the class default.
Real evaluation and training entry points¶
cp .env.example .env # fill in OPENROUTER_API_KEY or OPENAI_API_KEY, and MODEL
kind create cluster --config AIOpsLab/kind/kind-config-x86.yaml
cd AIOpsLab/aiopslab && cp config.yml.example config.yml && cd ../.. # set k8s_host: kind
python -m environment.aiopslab_server & # environment server, real serving command below
python -m main_aiopslab --problem k8s_target_port-misconfig-detection-1 # single task
python -m main_aiopslab # DEV_EVALUATE_ALL sweep over the 86-task list
README.md's own Quick Start section (re-read from the pinned clone) offers ./start_all.sh as a one-line alternative to the three commands above; that file does not exist anywhere in the repository tree at commit 17c8f55a0 (find . -iname "start_all*" returns nothing outside .git), so use the step-by-step form.
main_aiopslab.py's real argparse surface (async def main()) is --api-source {openrouter,openai}, --api-key, --api-base, --model, --host/--port (environment-server address), --problem / --problems (one or many task IDs) / --all, --session (resume an existing session ID), --output, --retries, --temperature, --max-context-tokens (default 25000), --max-output-tokens (default 8000); a DEV_MODE flag at the top of the file switches between this CLI and an in-file DEV_SPECIFIC_PROBLEMS list (the same 86 AIOpsLab task IDs eval_tasks.yaml-style benchmarks use) for local iteration. environment/aiopslab_server2.py's ensure_kind_cluster_and_kubeconfig() genuinely automates cluster bring-up: it runs kind export kubeconfig --kubeconfig <path>, sets KUBECONFIG in-process, and verifies with kubectl cluster-info, the same "whatever the exported kubeconfig already grants" access model as this KB's STRATUS page documents; no dedicated Kubernetes ServiceAccount/Role/RoleBinding ships in this repository either. This call runs unconditionally at module-import time, before the process even knows whether it is running on a laptop or inside a Pod; "The in-cluster credential guard" below patches it so it only bootstraps a Kind cluster in explicit local-dev mode, and relies purely on the Pod's own ServiceAccount otherwise.
The real serving command and HTTP surface¶
environment/aiopslab_server.py (README's documented module; aiopslab_server2.py is a near-duplicate that only changes two argparse defaults) is a real FastAPI application, not a stub: app = FastAPI(...) at line 1096, served by uvicorn.run(app, host=ServerConfig.HOST, port=ServerConfig.PORT, log_level="info") at line 1313. The genuine serving command is:
python -m environment.aiopslab_server --host 0.0.0.0 --port 8002
# flags read directly from the argparse block at the top of the file:
# --port (default 8002), --host (default 127.0.0.1), --cluster-name (default "kind"),
# --kind-config, --auto-delete
Every route below is a real @app.get/@app.post/@app.delete handler read directly from the file, not inferred from the README:
| Method | Path | Line | Purpose |
|---|---|---|---|
| GET | / |
1115 | server info: active sessions, cluster name |
| POST | /init_problem |
1138 | start an AIOpsLab problem instance for a session |
| POST | /execute_action |
1153 | run one Probe/Executor action against the live cluster |
| POST | /checkpoint |
1168 | snapshot cluster state for this session (see rollback below) |
| POST | /rollback |
1183 | restore a prior checkpoint (see rollback below) |
| GET | /session/{id}/status |
1198 | session status |
| DELETE | /session/{id} |
1210 | tear down a session |
| GET | /health |
1222 | liveness/readiness signal (used as the Deployment probe below) |
| POST | /port/cleanup |
1233 | force-clean stale port-forwards |
| GET | /port/status |
1246 | port-forward status for 32000-32009 |
| GET | /session/{id}/submit_format |
1264 | submission schema for the active problem |
| POST | /submit |
1276 | submit a solution for evaluation |
GET /health (line 1222) returns {"status": "healthy", "server_initialized": ..., "active_sessions": ..., "cluster_name": ...}; this is the only endpoint this repository ships that is suitable as a Kubernetes liveness/readiness probe, and it is what the Deployment manifest below actually probes.
Observer and Evolver GRPO training are two separate, real entry points (grpo/observer/train_grpo.py, grpo/evolver/train_grpo.py/train_grpo_trl.py), quoted from the README's own worked examples:
# Evolver GRPO, TRL-based trainer (recommended), single GPU
python grpo/evolver/train_grpo_trl.py \
--seed-dir data/gt/gt_c/claude-sonnet-4.5 --model Qwen/Qwen3-14B \
--reward-model anthropic/claude-sonnet-4.5 --batch-size 2 --num-generations 4 --num-epochs 3
# Evolver GRPO, custom trainer, more fine-grained control
python grpo/evolver/train_grpo.py \
--seed-dir data/gt/gt_c/claude-sonnet-4.5 --policy-model Qwen/Qwen2.5-7B-Instruct \
--reward-model anthropic/claude-sonnet-4-20250514 --group-size 4 --batch-size 2 \
--learning-rate 1e-5 --num-epochs 3 --use-lora --lora-rank 64
# Observer GRPO
python grpo/observer/train_grpo.py \
--policy-model-path Qwen/Qwen3-14B --reward-model anthropic/claude-sonnet-4.5 \
--vllm-gpu-memory 0.4 --vllm-max-model-len 14000
Real config classes, executed directly (no GPU required to validate the schema)¶
ObserverGRPOConfig (grpo/observer/grpo_config.py) is a plain dataclass; it and its group_size validation were constructed directly in this sandbox with no GPU or training run needed:
>>> from observer.grpo_config import ObserverGRPOConfig
>>> c = ObserverGRPOConfig()
>>> c.effective_batch_size # batch_size(2) * gradient_accumulation_steps(4) * group_size(4)
32
>>> c.group_size, c.num_epochs, c.lora_rank, c.lora_alpha
(4, 5, 64, 128)
>>> sum(c.reward_weights.get_all_weights().values())
1.0
>>> ObserverGRPOConfig(group_size=1)
ValueError: group_size must be at least 2 for GRPO
This is a real, executed repository-versus-paper gap, not asserted from prose: the paper's Section 5.1.3 states GRPO training uses "batch size 16, 3 epochs"; the shipped ObserverGRPOConfig default computes an effective batch size of 32 (2 x 4 x 4) and num_epochs=5. grpo/evolver/grpo_config.py's own default, by contrast, is num_epochs=3 and the same batch_size=2/group_size=4 shape, matching the paper's epoch count for that component even though Observer's does not. Re-derive whichever hyperparameters matter for your reproduction from the actual config file for the component you are training, not from Table 5 alone. One claim did check out exactly: grpo/observer/reward_model.py's JSON-parse-failure hard penalty is total_score = 0.09 verbatim, matching Eq. 3's stated R=0.09 penalty precisely.
The real Executor gate¶
agents/executor_agent.py's ExecutorAgent.__init__ takes available_actions: Dict[str, str], populated from AIOpsLab's orchestrator.init_problem() return value (the _apis this page's Architecture section already documents Observer never sees directly). This is the actual mechanism that limits what the Executor can call: the set of API actions AIOpsLab registers for the specific problem instance, not a static list of allowed kubectl command patterns. A repository-wide search for "whitelist" (grep -rln "whitelist\|allowlist\|allowed_command" --include="*.py" --include="*.yaml" .) returns zero matches anywhere in the tree, including inside the AIOpsLab submodule pinned at a56bb5db5d28348dba2ea66ae7693c0b4ee6e6ac. Do not build a production deployment assuming a 47-pattern command whitelist exists to audit; audit available_actions for the problem/task registration your deployment actually uses instead, and add your own command-level filtering if you need the write-side guardrail Table 5 describes.
available_actions is itself two layers, both read directly from the pinned AIOpsLab submodule rather than assumed from its docstrings. Layer 1 is a method-registry gate: MitigationTask.perform_action() (AIOpsLab/aiopslab/orchestrator/tasks/mitigation.py:66-72, the same pattern in detection.py, localization.py, and analysis.py) does getattr(self.actions, action_name, None) and raises InvalidActionError (AIOpsLab/aiopslab/utils/status.py:20-22) if the name is not a method on that task's Actions class; get_actions() (AIOpsLab/aiopslab/utils/actions.py:51-79) builds the exact same available_actions dict shown to the Executor's LLM by walking dir(class_obj) for every method decorated @action/@read/@write, so the registry and the enforcement gate are provably the same set. Layer 2 only applies to one specific registered action: exec_shell(command, timeout=30) (AIOpsLab/aiopslab/orchestrator/actions/base.py:79-107) runs the given string through Shell.exec(command) unmodified, filtered by nothing but a 5-entry substring BLOCK_LIST (kubectl edit, edit svc, kubectl port-forward, docker logs -f, kubectl logs -f). exec_shell is defined on the shared TaskActions base class, so get_actions()'s dir() walk registers it for every task type, Mitigation included; a Mitigation task's available_actions for a real problem is {submit, get_logs, exec_shell, get_metrics, read_metrics, get_traces, read_traces}, confirmed by reading MitigationActions(TaskActions) in AIOpsLab/aiopslab/orchestrator/actions/mitigation.py, which adds only submit() on top of the inherited base set. In practice this means the Executor's write path is not meaningfully allowlisted by command content at all: any shell command that avoids those 5 substrings runs, including ones with no relationship to the "kubectl rollout restart / scale / delete pod" pattern this page's earlier revisions used to illustrate the gate. The executed reproduction below demonstrates this gap directly against the real denylist logic, not a paraphrase of it.
Published checkpoints and training data¶
The README links real, released Hugging Face artifacts rather than only describing the pipeline: seed data (spacezenmasterr/aoi-planner-seeds-sonnet, spacezenmasterr/aoi-observer-training-data) and two LoRA checkpoints (spacezenmasterr/aoi-evolver-lora-ckpt490, spacezenmasterr/aoi-observer-lora-ckpt200). Seed data is JSON per successfully-resolved task (task_info, commands, evaluation_results) under data/gt/gt_c/<model>/<task_id>.json; train_grpo_trl.py supports --resume-from-checkpoint <dir> (continue an interrupted run) and --load-weights-from <dir> (start fresh training from a prior checkpoint's weights, a different operation from resume), both real flags in the training CLI. trainer.save_model(config.checkpoint_dir) is the real checkpoint write path, with save_steps/save_total_limit from ObserverGRPOConfig controlling cadence and retention.
When adapting the benchmark's 26 fault types to a real estate, mirror the paper's nested, leakage-free data split rather than a random holdout: build the Evolver's training set from every successfully-resolved historical incident, take a strict fault-type subset of that (zero type overlap with your held-out evaluation set) for Observer GRPO, and keep every test case unseen by both components, exactly as the paper enforces Dobs_train subset of Devolver_train subset of Dall so the combined system's evaluation carries no leakage between the two trained pieces.5
How to develop and extend it¶
- Reward dimensions are where behavior gets tuned, not the base model. Context Instruction and Context Namespace carry 60% of the Observer's step reward by design, reflecting the authors' bet that diagnostic reasoning quality and target accuracy dominate information gain per step; Format is rule-based (a JSON parse failure is a hard
R=0.09penalty regardless of the other five scores) so malformed output can never win a GRPO group by accident.3 If you extend the reward, keep a rule-based floor for structural validity separate from the LLM-judged dimensions, the same separation the executed reward function below enforces. - Seed provenance is deliberately swappable. The paper uses Claude Sonnet 4.5 trajectories on AIOpsLab as a stand-in for expert SRE records purely because that model provided enough high-quality successes on the benchmark; the Evolver's design does not care whether seeds come from a frontier model, human runbooks, or the system's own historical successes.4 Point the Purifier and Evolver at your own historical incident tickets once you have enough resolved ones, and success-vs-failure classification (the Judge in Figure 1) becomes the only part that needs a human-in-the-loop sign-off.
- The Evolver's own stated limitation is a good next-extension target. Section 7 flags that the Evolver currently only emits corrected command sequences as static prompts; the authors call out producing synthetic environment feedback via a simulator, or running the Evolver as a live runtime agent for dynamic plan refinement, as unexplored future architecture, not something the current release does.10
- Localization is the component that most needs task-aware tuning before you trust GRPO broadly. Because the same reward that improves Detection degrades Localization (Appendix D.4's root-cause analysis: over-exploration surfaces multiple anomaly candidates and the trained model picks the wrong one), a task-type-conditioned reward or a separate exploration budget per task type is the change the paper's own analysis points toward but does not implement.9
How to run it in production¶
Treat AOI the way this KB treats any pre-production agent framework: as a candidate to gate against your own incident history before it gets write access, not as a deployed reference. Three production-shaped facts from the paper matter for that gate. First, the read-write separation is not a compliance tax: the paper's own framing is that constraining the action space improved diagnostic success rather than reducing it, because evidence accumulation before mutation avoided the cascading state corruption STRATUS exhibited when it mutated before diagnosis completed.10 Second, GRPO training on 23 tasks generalized to 15 unseen fault types without collapsing, but it did so unevenly across task types, so any production rollout should track success by task type (Detection/Localization/RCA/Mitigation), not a single blended accuracy number, exactly because the blended 42.9% overall hides a Localization regression.7 Third, budget planning should use the paper's own diminishing-returns curve: best@1 is 31.4%, best@2 is 51.2% (a single retry captures most of the easy wins, +19.8pp), and gains flatten sharply after that (best@3 58.1%, best@5 66.3%); the paper recommends k=2 sampling rounds for cost-sensitive deployments and k=3 (which captures 88% of the achievable improvement) for higher-stakes ones, rather than assuming more samples keeps paying off.9
Three concrete gaps in the released repository need closing before any real write access, confirmed by reading the source rather than assumed from the paper. First, there is no Kubernetes RBAC manifest anywhere in the tree; environment/aiopslab_server2.py exports a kind cluster's admin kubeconfig and sets KUBECONFIG in-process, so the Executor's write access in this reference setup is whatever that kubeconfig grants, cluster-admin by default, not a scoped ServiceAccount. Bind the process to a namespace-scoped Role/RoleBinding before pointing it at anything beyond a disposable test cluster; a real, server-validated manifest that does this is below rather than left as prose. Second, and closely related: even with that Role/RoleBinding bound to the Pod, the server code itself overrides the Pod's own credentials with a Kind cluster's admin kubeconfig at import time, so binding a scoped ServiceAccount is not sufficient by itself, the server has to be patched to stop clobbering it; "The in-cluster credential guard" below closes this with a real patch, applied to a real image, and validated with a SelfSubjectAccessReview executed from inside the running Pod. Third, do not audit the Executor's guardrails by looking for a "47-pattern whitelist"; none exists in this repository (see "The real Executor gate" above). The actual write-side constraint is AIOpsLab's per-problem available_actions registry enforced by a getattr-based method gate, plus a 5-entry substring denylist scoped only to the exec_shell action (see "Executed: AIOpsLab's real two-layer command gate" above); that denylist blocks four specific kubectl/docker footguns and nothing else, including in-process secret reads and command injection, so a deployment against your own estate needs its own real command-level filter layered on top, not an assumption that available_actions already provides one, the same lesson this KB's STRATUS page draws from finding a construction bug in that project's own linter; "A real allowlist-based tightening of the exec_shell gate" below closes the specific bypasses found. For checkpoint rollout, the published LoRA adapters (spacezenmasterr/aoi-observer-lora-ckpt200, spacezenmasterr/aoi-evolver-lora-ckpt490) are a real starting point for a canary: serve the base Qwen3-14B plus adapter behind vLLM, compare Detection/Localization/RCA/Mitigation success by task type against the untrained baseline on a held-out slice of your own incident history (mirroring the paper's Dobs_test split) before promoting the adapter to the primary serving path, and keep the pre-adapter base-model serving path available as the rollback target since trainer.save_model writes adapters, not merged weights, so reverting is a config change, not a model surgery.
The in-cluster credential guard: closing the KUBECONFIG-at-import defect¶
environment/aiopslab_server.py (the module the Deployment below actually runs; aiopslab_server2.py is patched identically) calls ensure_kind_cluster_and_kubeconfig(args.cluster_name, args.kind_config) unconditionally at module-import time (line 116 of the pinned clone, before the FastAPI app object even exists), and EnvironmentServer.startup() calls create_kind_cluster() again on every FastAPI startup event. Both shell out to the kind binary; both, on success, overwrite os.environ['KUBECONFIG'] and ~/.kube/config with a fresh Kind cluster's admin kubeconfig. Two real consequences follow, verified in this sandbox rather than assumed from reading the code: first, kind is not installed in the serving image built below, so subprocess.run(["kind", "--version"], ...) raises FileNotFoundError, which ensure_kind_cluster_and_kubeconfig() catches and turns into sys.exit(1), so the unpatched server cannot even finish importing inside a Pod. Second, on any host where kind and Docker-in-Docker were reachable from the Pod (a materially worse outcome), the Pod's own scoped ServiceAccount would simply be discarded in favor of cluster-admin, defeating the RBAC manifest below entirely.
The fix, applied to a real clone at the pinned commit and confirmed with python3 -m py_compile before building: detect in-cluster execution the same way kubernetes.config.load_incluster_config() does (KUBERNETES_SERVICE_HOST is set by kubelet for every Pod, and the projected ServiceAccount token is always mounted), and only run the Kind-bootstrap path when an explicit AOI_BOOTSTRAP_KIND=1 flag is set. When in-cluster and that flag is absent, a small helper builds a kubeconfig directly from the Pod's own mounted token, CA cert, and KUBERNETES_SERVICE_HOST/PORT, the file-based equivalent of load_incluster_config(); this is necessary rather than just relying on the in-cluster loader because this codebase's own observer/__init__.py and service/kubectl.py call the file-based config.load_kube_config() (one of them with a hardcoded config_file='~/.kube/config' that ignores KUBECONFIG entirely), and some code shells out to the kubectl CLI directly, which has no built-in in-cluster mode at all. The generated file is written to both the KUBECONFIG path and ~/.kube/config, and its context is named kind-{AIOPSLAB_CLUSTER} (default kind-kind) to match kubectl.py's own hardcoded context lookup, so no other call site needs patching:
+_SA_DIR = Path("/var/run/secrets/kubernetes.io/serviceaccount")
+IN_CLUSTER = bool(os.environ.get("KUBERNETES_SERVICE_HOST")) and (_SA_DIR / "token").exists()
+BOOTSTRAP_KIND = os.environ.get("AOI_BOOTSTRAP_KIND") == "1"
+
+def _write_incluster_kubeconfig() -> str:
+ """Build a kubeconfig from the Pod's own mounted ServiceAccount token,
+ CA cert, and the kubelet-injected KUBERNETES_SERVICE_HOST/PORT -- the
+ file-based equivalent of load_incluster_config(). Never touches Kind
+ or any admin credential. Context named kind-{AIOPSLAB_CLUSTER} to match
+ kubectl.py's own hardcoded context lookup."""
+ ... # writes /tmp/aoi-in-cluster-kubeconfig.yaml AND ~/.kube/config
-# 立即执行集群创建和kubeconfig生成
-ensure_kind_cluster_and_kubeconfig(args.cluster_name, args.kind_config)
+if IN_CLUSTER and not BOOTSTRAP_KIND:
+ os.environ["KUBECONFIG"] = _write_incluster_kubeconfig()
+elif BOOTSTRAP_KIND:
+ ensure_kind_cluster_and_kubeconfig(args.cluster_name, args.kind_config)
+else:
+ print("[local-dev] leaving KUBECONFIG untouched; set AOI_BOOTSTRAP_KIND=1 "
+ "to auto-create or reuse a local Kind cluster.")
The same IN_CLUSTER gate also replaces the unconditional create_kind_cluster() call inside EnvironmentServer.startup(). Both files (aiopslab_server.py and aiopslab_server2.py) were patched identically and confirmed to compile (python3 -m py_compile environment/aiopslab_server.py environment/aiopslab_server2.py) before the image below was built; this diff is abbreviated for readability, the full patch is under 90 lines per file.
Correctness was verified twice, in increasing order of realism. First, a synthetic in-cluster check: docker run with KUBERNETES_SERVICE_HOST/PORT set and a fake ServiceAccount directory (token, ca.crt, namespace) bind-mounted at /var/run/secrets/kubernetes.io/serviceaccount produced Environment Server ready at http://0.0.0.0:8002 with no traceback, and the generated /tmp/aoi-in-cluster-kubeconfig.yaml and ~/.kube/config both matched the mounted token and KUBERNETES_SERVICE_HOST/PORT exactly. Second, the real cluster test in the next section: the Pod's actual, kubelet-issued ServiceAccount token, decoded from the generated kubeconfig, carries "sub":"system:serviceaccount:hotel-res:aoi-executor", confirming the running process really is using the scoped identity, not an admin one.
Separate platform bootstrap from problem initialization¶
The 403 reproduced below is not safely fixed by granting the long-running AOI Pod list pods in openebs. At the pinned AIOpsLab commit, Orchestrator.init_problem() first runs kubectl apply for the full OpenEBS operator and patches a cluster-scoped StorageClass; KubeCtl.exec_command() converts any nonzero subprocess exit into a returned stderr string instead of raising, so both authorization failures are silently ignored. wait_for_ready("openebs") then performs the first Kubernetes-client call that raises, CoreV1Api.list_namespaced_pod("openebs"), producing the visible 403. Adding only that read permission would leave OpenEBS absent and turn the 403 into a 300-second readiness timeout. Granting the operator install, StorageClass patch, CRDs, and namespace lifecycle to this service would recreate cluster-admin in pieces.
The least-privilege fix is a code-path split: a platform administrator pre-provisions OpenEBS, Prometheus, and the target namespace; the in-cluster AOI process only initializes and mutates the selected benchmark application inside that namespace. The following patch was compiled against AIOpsLab commit a56bb5db5d28348dba2ea66ae7693c0b4ee6e6ac and exercised through Orchestrator.init_problem() with fakes only at the Kubernetes and Prometheus boundaries:
class Orchestrator:
def __init__(self, results_dir=None):
...
+ self.platform_mode = os.getenv("AIOPSLAB_PLATFORM_MODE", "bootstrap")
+ if self.platform_mode not in {"bootstrap", "preprovisioned"}:
+ raise ValueError("AIOPSLAB_PLATFORM_MODE must be bootstrap or preprovisioned")
- if deployment != "docker":
+ if deployment != "docker" and self.platform_mode == "bootstrap":
# existing OpenEBS install + StorageClass patch
...
self.prometheus = Prometheus()
self.prometheus.deploy()
+ elif deployment != "docker":
+ print("Using preprovisioned OpenEBS and Prometheus; "
+ "skipping cluster-scoped platform bootstrap.")
- self.session.problem.app.cleanup()
+ if self.platform_mode == "bootstrap":
+ self.session.problem.app.cleanup() # may delete namespace/PVs
+ else:
+ self.session.problem.app.delete() # target manifests only
- if self.session.problem.namespace != "docker":
+ if (self.session.problem.namespace != "docker"
+ and self.platform_mode == "bootstrap"):
# existing Prometheus/OpenEBS teardown
...
+ elif self.session.problem.namespace != "docker":
+ print("Leaving preprovisioned OpenEBS and Prometheus running.")
The default remains bootstrap, preserving the upstream local Kind workflow. Production must set preprovisioned explicitly; any other value fails before initialization. Executed regression output:
preprovisioned: skipped OpenEBS/Prometheus; target deploy and fault path ran
bootstrap: original OpenEBS/Prometheus path retained
invalid mode: rejected before initialization
The preprovisioned branch removes every openebs API call from the AOI identity; it does not pretend OpenEBS or Prometheus are optional. Install and health-check them with the platform deployment pipeline before enabling this Deployment. This page does not grant the AOI ServiceAccount access to either platform namespace.
A real, least-privilege RBAC manifest (server-validated, not just described)¶
The server stays in the hotel-res controller namespace, but the pinned AIOpsLab Hotel Reservation metadata names its actual application namespace test-hotel-reservation. A RoleBinding in that target namespace can bind the hotel-res/aoi-executor ServiceAccount without granting anything cluster-wide. The Role covers the exact pinned application manifest kinds (Deployment, Service, PersistentVolumeClaim), the ConfigMaps HotelReservation.__init__() creates, Pod readiness, log and delete operations, and checkpoint/rollback reads and applies. It does not grant Secrets, namespaces, nodes, OpenEBS, or the observe Prometheus namespace. Pre-create test-hotel-reservation in the platform pipeline.
# rbac.yaml -- controller ServiceAccount in hotel-res; namespaced permissions
# only in the pre-created test-hotel-reservation target. No cluster-admin or
# OpenEBS/Prometheus bootstrap permissions.
apiVersion: v1
kind: ServiceAccount
metadata:
name: aoi-executor
namespace: hotel-res
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: aoi-target-controller
namespace: test-hotel-reservation
rules:
- apiGroups: ["apps"]
resources: ["deployments", "replicasets", "statefulsets"]
verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
- apiGroups: [""]
resources: ["pods", "pods/log", "configmaps", "services", "events", "persistentvolumeclaims"]
verbs: ["get", "list", "watch", "create", "patch", "update", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: aoi-target-controller-binding
namespace: test-hotel-reservation
subjects:
- kind: ServiceAccount
name: aoi-executor
namespace: hotel-res
roleRef:
kind: Role
name: aoi-target-controller
apiGroup: rbac.authorization.k8s.io
# deployment.yaml -- binds the Executor process to the scoped ServiceAccount
# above instead of whatever KUBECONFIG happens to be on the host. USE_WANDB,
# AIOPSLAB_USE_PROBLEM_VARIANTS, and OPENROUTER_* come from the upstream
# environment contract; AIOPSLAB_PLATFORM_MODE selects the patch above.
# Command, arguments, and probes use the real serving surface documented above.
apiVersion: apps/v1
kind: Deployment
metadata:
name: aoi-executor
namespace: hotel-res
spec:
replicas: 1
selector:
matchLabels:
app: aoi-executor
template:
metadata:
labels:
app: aoi-executor
spec:
serviceAccountName: aoi-executor
containers:
- name: aoi-executor
image: aoi-executor:local
imagePullPolicy: Never # loaded locally via `kind load docker-image`, never pulled from a registry
command: ["python", "-m", "environment.aiopslab_server"]
args: ["--host", "0.0.0.0", "--port", "8002"]
ports:
- containerPort: 8002
name: http
readinessProbe:
httpGet:
path: /health
port: 8002
initialDelaySeconds: 10
periodSeconds: 10
livenessProbe:
httpGet:
path: /health
port: 8002
initialDelaySeconds: 20
periodSeconds: 20
env:
- name: USE_WANDB
value: "false"
- name: AIOPSLAB_PLATFORM_MODE
value: "preprovisioned"
- name: AIOPSLAB_USE_PROBLEM_VARIANTS
value: "true"
- name: OPENROUTER_BASE_URL
value: "https://openrouter.ai/api/v1"
- name: OPENROUTER_MODEL
value: "openai/gpt-4o-mini"
- name: OPENROUTER_API_KEY
valueFrom:
secretKeyRef:
name: aoi-executor-secrets
key: openrouter-api-key
resources:
requests:
cpu: "250m"
memory: "512Mi"
limits:
cpu: "1"
memory: "2Gi"
# service.yaml -- ClusterIP in front of the FastAPI /health, /execute_action,
# /checkpoint, and /rollback endpoints documented above; selector matches the
# Deployment's pod-template labels.
apiVersion: v1
kind: Service
metadata:
name: aoi-executor
namespace: hotel-res
spec:
selector:
app: aoi-executor
ports:
- name: http
port: 8002
targetPort: 8002
type: ClusterIP
All three manifests were first validated against a real kind cluster created for this check (kind create cluster --name aoi-verify, kubectl client v1.34.5, server v1.31.0), not just checked for YAML syntax, with kubectl apply --dry-run=server followed by a real apply; that cluster was deleted afterward.
Getting the Deployment to Running¶
The RBAC/Deployment/Service manifests above were re-applied against a second, fresh, disposable cluster (kind create cluster --name aoi-kb-fix, kubectl client v1.34.5, server v1.31.0, later deleted) built with the patched source from "The in-cluster credential guard" above, to close the ImagePullBackOff finding for real rather than re-describe it. The image was built from the pinned clone with the two files patched, python:3.11-slim, requirements.txt installed with pip install --no-deps (the pinned set has an unresolvable conflict between aworld==0.2.6's aiohttp~=3.9.5 and auth0-python's aiohttp>=3.10.11; the AIOpsLab submodule's own pip install -e . step still fails on a missing setuptools in the build stage and is tolerated with || true, matching how README.md itself installs it as a separate, non-blocking step), producing a real ~2 GB aoi-executor:local image, not a placeholder:
$ docker build -t aoi-executor:local .
...
#12 exporting to image
#12 writing image sha256:3033e07c1a40954412738f550b5ceab112b1f92ab9e9249e7db8134b61105e14 done
#12 naming to docker.io/library/aoi-executor:local done
$ kind load docker-image aoi-executor:local --name aoi-kb-fix
Image: "aoi-executor:local" with ID "sha256:3033e07c1..." not yet present on node "aoi-kb-fix-control-plane", loading...
$ kubectl apply -f rbac.yaml && kubectl apply -f secret.yaml && kubectl apply -f deployment.yaml && kubectl apply -f service.yaml
serviceaccount/aoi-executor created
role.rbac.authorization.k8s.io/aoi-executor-role created
rolebinding.rbac.authorization.k8s.io/aoi-executor-binding created
secret/aoi-executor-secrets created
deployment.apps/aoi-executor created
service/aoi-executor created
$ kubectl get pods -n hotel-res -o wide
NAME READY STATUS RESTARTS AGE IP NODE
aoi-executor-6654698fd9-7dmnk 1/1 Running 0 27s 10.244.0.5 aoi-kb-fix-control-plane
$ kubectl logs -n hotel-res -l app=aoi-executor --tail=20
[in-cluster] Using the Pod's scoped ServiceAccount token as KUBECONFIG (/tmp/aoi-in-cluster-kubeconfig.yaml); skipped Kind bootstrap and never touched an admin kubeconfig.
[in-cluster] Skipping Kind cluster setup at startup; already using the Pod's ServiceAccount credentials.
✅ Environment Server ready at http://0.0.0.0:8002
INFO: 10.244.0.1:37004 - "GET /health HTTP/1.1" 200 OK
1/1 Running with a passing /health readiness probe, not ImagePullBackOff; the log lines confirm the in-cluster guard is the code path that actually ran. The generated kubeconfig's token decodes to "sub":"system:serviceaccount:hotel-res:aoi-executor", not cluster-admin. The corrected cross-namespace Role was then applied to a fresh Kind cluster and checked against that exact identity:
$ SA=system:serviceaccount:hotel-res:aoi-executor
$ kubectl auth can-i create deployments -n test-hotel-reservation --as=$SA
yes
$ kubectl auth can-i delete persistentvolumeclaims -n test-hotel-reservation --as=$SA
yes
$ kubectl auth can-i create configmaps -n test-hotel-reservation --as=$SA
yes
$ kubectl auth can-i patch services -n test-hotel-reservation --as=$SA
yes
$ kubectl auth can-i list pods -n test-hotel-reservation --as=$SA
yes
$ kubectl auth can-i get secrets -n test-hotel-reservation --as=$SA
no
$ kubectl auth can-i list pods -n openebs --as=$SA
no
$ kubectl auth can-i delete namespaces --as=$SA
no
$ kubectl auth can-i list nodes --as=$SA
no
The Role admits target application initialization and rollback-shaped namespaced operations while continuing to deny the exact OpenEBS call that exposed the unsafe bootstrap coupling. The Service boundary was separately confirmed against the real serving Pod: a client Pod resolved the Service and received /health through ClusterIP.
$ kubectl run curl-check -n hotel-res --image=curlimages/curl:8.10.1 --restart=Never --command -- \
sh -c "curl -s http://aoi-executor.hotel-res.svc.cluster.local:8002/health"
{"status":"healthy","server_initialized":true,"active_sessions":0,"cluster_name":"kind"}
A real rollback command¶
Two independent rollback paths exist for this deployment shape.
Kubernetes-level rollback, for a bad image or config push: kubectl rollout undo reverts a Deployment to its previous revision. Demonstrated end to end against the genuinely-Running Deployment from "Getting the Deployment to Running" above, including a real bad rollout first and a health check confirming the restored Pod actually serves traffic again, not just that the image field reverted:
$ kubectl rollout history deployment/aoi-executor -n hotel-res
REVISION CHANGE-CAUSE
1 <none>
$ kubectl get pods -n hotel-res
NAME READY STATUS RESTARTS AGE
aoi-executor-6654698fd9-7dmnk 1/1 Running 0 109s
$ kubectl set image deployment/aoi-executor aoi-executor=aoi-executor:local-bad -n hotel-res
deployment.apps/aoi-executor image updated
$ kubectl get pods -n hotel-res
NAME READY STATUS RESTARTS AGE
aoi-executor-5f8d49b875-mdwc5 0/1 ErrImageNeverPull 0 8s
aoi-executor-6654698fd9-7dmnk 1/1 Running 0 118s
$ kubectl rollout undo deployment/aoi-executor -n hotel-res
deployment.apps/aoi-executor rolled back
$ kubectl rollout status deployment/aoi-executor -n hotel-res --timeout=60s
deployment "aoi-executor" successfully rolled out
$ kubectl get deployment aoi-executor -n hotel-res -o jsonpath='{.spec.template.spec.containers[0].image}'
aoi-executor:local
$ kubectl get pods -n hotel-res
NAME READY STATUS RESTARTS AGE
aoi-executor-6654698fd9-7dmnk 1/1 Running 0 2m29s
$ kubectl run curl-check2 -n hotel-res --image=curlimages/curl:8.10.1 --restart=Never --command -- \
sh -c "curl -s http://aoi-executor.hotel-res.svc.cluster.local:8002/health"
{"status":"healthy","server_initialized":true,"active_sessions":0,"cluster_name":"kind"}
The bad rollout produced a real, live ErrImageNeverPull (the tag does not exist and imagePullPolicy: Never forbids trying to pull it), not a simulated failure; kubectl rollout undo reverted the Deployment's pod template, and the pre-existing healthy replica (never terminated, since the new replica never became ready) continued serving, confirmed by the post-rollback health check succeeding through the Service.
Application-level rollback, for a bad in-session action rather than a bad deploy: the FastAPI server itself ships a real checkpoint/rollback pair, not a config-change euphemism. POST /checkpoint (line 1168) calls OrchestratorSession.create_checkpoint() (environment/aiopslab_server.py:752-770), which runs kubectl get all -o yaml -n <namespace> and stores the YAML to a temp file (Checkpoint.save_state, environment/aiopslab_server.py:317-349); POST /rollback (line 1183) calls rollback_to_checkpoint() (environment/aiopslab_server.py:773-808), which runs kubectl apply -f <that temp file> (Checkpoint.restore_state, environment/aiopslab_server.py:352-364) and truncates the session's action history back to the checkpoint. Invoked over HTTP:
curl -sX POST http://aoi-executor.hotel-res.svc.cluster.local:8002/checkpoint \
-H 'Content-Type: application/json' \
-d '{"session_id": "<id>", "checkpoint_name": "before-mitigation"}'
curl -sX POST http://aoi-executor.hotel-res.svc.cluster.local:8002/rollback \
-H 'Content-Type: application/json' \
-d '{"session_id": "<id>", "checkpoint_name": "before-mitigation"}'
Both /checkpoint and /rollback require an active session, which requires a successful POST /init_problem first. The real pre-fix attempt used k8s_target_port-misconfig-detection-1, a SocialNetwork task whose pinned metadata targets test-social-network, while the old Role targeted hotel-res. It reached platform bootstrap before target deployment and failed at OpenEBS readiness:
[16:23:01] Error checking pod statuses: (403)
Reason: Forbidden
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},
"status":"Failure","message":"pods is forbidden: User
\"system:serviceaccount:hotel-res:aoi-executor\" cannot list resource \"pods\"
in API group \"\" in the namespace \"openebs\"","reason":"Forbidden",...}
Source inspection shows why authorizing that single call is wrong: the immediately preceding OpenEBS apply and cluster-scoped StorageClass patch had already failed authorization, but KubeCtl.exec_command() returned stderr instead of raising. Pod-list permission would only hide the first useful error and poll an installation that never happened. The preprovisioned platform mode above removes that bootstrap and teardown path from this identity, and the corrected example task for this Role is misconfig_app_hotel_res-detection-1, whose real namespace is test-hotel-reservation.
The managed-platform patch and Role were validated independently, but the 2 GB image was not rebuilt and a full /init_problem was not rerun after this correction. Therefore neither /checkpoint nor /rollback is claimed as executed. The next honest integration gate is:
# Preconditions, owned by the platform pipeline: OpenEBS and Prometheus healthy;
# namespace/test-hotel-reservation created; corrected image loaded.
curl -fsS -X POST http://aoi-executor.hotel-res.svc.cluster.local:8002/init_problem \
-H 'Content-Type: application/json' \
-d '{"problem_id":"misconfig_app_hotel_res-detection-1","reset_if_exists":false}'
Only after that returns a session ID should the /checkpoint and /rollback calls above be attempted and their restored Kubernetes state compared. A health-only result remains insufficient.
Checkpoint-level rollback, for a bad LoRA adapter promoted to serving: trainer.save_model(config.checkpoint_dir) (documented above under "Published checkpoints and training data") writes adapter weights, not merged weights, so reverting a bad canary is a config change, not a model surgery: point the vLLM serving process's adapter path (or train_grpo_trl.py --load-weights-from) back at the prior checkpoint directory, e.g. spacezenmasterr/aoi-observer-lora-ckpt200 instead of a newly trained adapter, and restart serving. Neither vLLM nor a serving script for the published adapters ships in this repository; this is the rollback mechanism the checkpoint format supports, not a command this repository runs for you.
Monitoring the real serving path¶
Two real, distinct signals exist; do not conflate them. First, the FastAPI server's own health: GET /health (line 1222, used as the Deployment's readiness/liveness probe above) and the plain-text per-agent logs utils/logger_config.py's AgentLogger/FileLogHandler write to ./log/<model_name>[-round<N>]/<problem_id>.log (FileLogHandler.set_log_file, utils/logger_config.py:26-38; round selection via the ROUND env var, per-agent enable flags in AgentLogger.ENABLED_AGENTS). The server exposes no native /metrics Prometheus endpoint for its own process (confirmed by reading every @app. route in the file; only /, /health, and /port/status return status information) so scrape it as an HTTP blackbox probe against /health, or tail/ship the log directory with a sidecar; do not point a Prometheus scrape config at this service expecting an exposition format it does not produce. Second, and easily confused with the first: AIOpsLab's own Prometheus().deploy() (invoked from orchestrator.init_problem()) stands up a Prometheus instance inside the simulated target cluster (hotel-res, etc.) that the Executor's get_metrics/read_metrics actions query on port 32000; this monitors the fault-injected application under test, not the AOI serving process itself, and is unrelated to observing the Deployment above.
Executed: read-write separation and the Executor gate¶
This models Table 1's permission matrix, Algorithm 1's main loop, and the least-privilege gate the Introduction describes ("high-risk write commands are technically isolated and can only be triggered after sufficient evidence is gathered and verified"), plus the Executor's real action gate. An earlier revision of this page modeled that gate as a static "whitelist" set, in tension with this page's own finding, two sections up, that no whitelist artifact exists in the repository; the model below now uses AIOpsLab's actual per-problem available_actions registry instead (see "The real Executor gate" above), so the executed code no longer contradicts the prose next to it. It is still a model of the mechanism, not a run of the real Observer/Probe/Executor/Compressor agents.
# aoi_readwrite_gate.py - validated: AOI's read-write separated execution
# architecture (Table 1 permission matrix, Section 3.1 key invariants,
# Algorithm 1 main loop) and the Executor's REAL action gate: AIOpsLab's
# per-problem available_actions registry, shaped exactly like
# ExecutorAgent.__init__'s available_actions: Dict[str, str] parameter
# (agents/executor_agent.py), populated from orchestrator.init_problem()'s
# return value. Not the paper's unshipped "47-pattern whitelist" (Table 5);
# a repository-wide grep confirms no whitelist artifact exists at this
# commit (see "The real Executor gate" above). Models the mechanism; does
# not run AOI or Qwen3-14B.
from __future__ import annotations
from dataclasses import dataclass, field
from typing import Literal
Store = Literal["Mraw", "Mtask", "Mcomp"]
Mode = Literal["R", "W"]
# Table 1: agent x store access-control matrix ("-" = no access).
PERMISSIONS: dict[str, dict[Store, str]] = {
"Observer": {"Mraw": "-", "Mtask": "RW", "Mcomp": "R"},
"Probe": {"Mraw": "W", "Mtask": "R", "Mcomp": "-"},
"Executor": {"Mraw": "W", "Mtask": "R", "Mcomp": "-"},
"Compressor": {"Mraw": "R", "Mtask": "-", "Mcomp": "W"},
}
class PermissionError_(Exception):
pass
def access(agent: str, store: Store, mode: Mode) -> None:
"""Enforce Table 1: raise unless the agent's grant covers this mode."""
grant = PERMISSIONS[agent][store]
if mode not in grant:
raise PermissionError_(f"{agent} has no {mode} access to {store} (grant={grant!r})")
# The real gate, shaped like a single AIOpsLab problem's available_actions
# registry (orchestrator.init_problem()'s return value); no static whitelist
# exists in the repository to model instead.
AVAILABLE_ACTIONS: dict[str, str] = {
"kubectl rollout restart": "restart a deployment's pods",
"kubectl scale": "change a deployment's replica count",
"kubectl delete pod": "delete a pod so its controller recreates it",
}
def executor_run(cmd: str, evidence_gathered: bool) -> str:
"""Section 3.1 / Section 1: Executor 'operates under strict whitelists'
and high-risk write commands 'can only be triggered after sufficient
evidence is gathered and verified'. The real gate is per-problem
available_actions, not a static whitelist (see module docstring)."""
if not evidence_gathered:
raise PermissionError_("Executor blocked: no verified evidence yet (least-privilege gate)")
if cmd not in AVAILABLE_ACTIONS:
raise PermissionError_(f"Executor blocked: {cmd!r} not in this problem's available_actions")
return f"executed: {cmd}"
@dataclass
class Stores:
raw: dict = field(default_factory=dict)
task: dict = field(default_factory=dict)
comp: dict = field(default_factory=dict)
def compress(raw_n: dict) -> dict:
"""Compressor: 'stateless, processing each iteration independently to
avoid error accumulation' (Sec 3.1, invariant 3). Pure function of the
CURRENT iteration's raw output only, no closure over prior calls."""
return {"summary": f"{len(raw_n)} raw records", "n_records": len(raw_n)}
def run_loop(max_iterations: int, decisions: list[tuple[str, dict]]) -> tuple[list[dict], str]:
"""Algorithm 1: Observer decides an action in {Probe, Execute, Submit}
each iteration; only Mcomp reaches the Observer (invariant 1); budget
exhaustion (Table 5: max_iterations=15) submits a timeout, never hangs."""
stores = Stores()
history: list[dict] = []
evidence_gathered = False
for n, (action, payload) in enumerate(decisions[:max_iterations], start=1):
access("Observer", "Mcomp", "R") # Observer only ever reads Mcomp
if action == "Submit":
return history, "submitted"
if action == "Probe":
access("Probe", "Mraw", "W")
stores.raw[n] = payload
evidence_gathered = True
elif action == "Execute":
access("Executor", "Mraw", "W")
result = executor_run(payload["cmd"], evidence_gathered)
stores.raw[n] = {"result": result}
c_n = compress(stores.raw[n])
access("Compressor", "Mcomp", "W")
stores.comp[n] = c_n
history.append(c_n)
return history, "budget_exhausted"
# 1. Legal path: Probe gathers evidence, then a registered Execute succeeds,
# then Submit. Table 5 caps this at max_iterations=15.
decisions = [("Probe", {"q": "kubectl get pods"}),
("Execute", {"cmd": "kubectl rollout restart"}),
("Submit", {})]
hist, status = run_loop(max_iterations=15, decisions=decisions)
assert status == "submitted" and len(hist) == 2
# 2. Adversarial: Observer must never read Mraw directly, even if a caller
# tries to route around Mcomp (invariant 1, the basis for the paper's
# "Observer never directly interacts with the environment" claim).
try:
access("Observer", "Mraw", "R")
assert False, "Observer must not be able to read Mraw"
except PermissionError_:
pass
# 3. Adversarial: Probe writes raw but cannot read it back, all information
# must flow back through the Compressor (invariant 2).
access("Probe", "Mraw", "W")
try:
access("Probe", "Mraw", "R")
assert False, "Probe must not be able to read Mraw back"
except PermissionError_:
pass
# 4. Adversarial: least-privilege gate. Execute before any Probe evidence is
# denied even for a registered command.
try:
executor_run("kubectl scale", evidence_gathered=False)
assert False, "Executor must not fire before evidence is gathered"
except PermissionError_:
pass
# 5. Adversarial: a command outside this problem's available_actions is
# rejected even with evidence in hand. There is no static "47-pattern
# whitelist" (Table 5) to consult; see "The real Executor gate" above.
try:
executor_run("rm -rf /", evidence_gathered=True)
assert False, "a command outside available_actions must be rejected"
except PermissionError_:
pass
# 6. Budget exhaustion: an Observer that never submits is cut off at
# max_iterations and does not hang (Algorithm 1, line 19).
never_submits = [("Probe", {"q": f"probe-{i}"}) for i in range(20)]
hist2, status2 = run_loop(max_iterations=15, decisions=never_submits)
assert status2 == "budget_exhausted" and len(hist2) == 15
# 7. Compressor statelessness: compressing same-size raw payloads, with an
# unrelated intervening call, must be independent of call history
# (invariant 3, no error accumulation across iterations).
c_a = compress({"x": 1, "y": 2})
compress({"unrelated": "call", "with": "different", "size": 4})
c_b = compress({"p": 9, "q": 8})
assert c_a["n_records"] == c_b["n_records"] == 2
print("legal path status:", status, "| iterations:", len(hist))
print("budget-exhausted after", len(hist2), "of", len(never_submits), "attempted iterations")
print("all read-write separation and available-actions-gate assertions passed")
Executed output:
legal path status: submitted | iterations: 2
budget-exhausted after 15 of 20 attempted iterations
all read-write separation and available-actions-gate assertions passed
Executed: AIOpsLab's real two-layer command gate¶
The block above models available_actions as a flat allowlist of command strings for clarity. The real gate, reproduced here verbatim from the pinned AIOpsLab submodule ("The real Executor gate" above), is two layers and neither one is a 47-pattern allowlist: a method-registry check (getattr against the task's Actions class) and, only for the exec_shell action, a 5-entry substring denylist. This is executed, not paraphrased, including the adversarial case the denylist does not catch.
# aoi_command_filter.py - validated: AIOpsLab's REAL two-layer command gate,
# reproduced from the submodule pinned at a56bb5db5d28348dba2ea66ae7693c0b4ee6e6ac
# (the exact commit AOI's own pinned commit 17c8f55a0 references via .gitmodules).
# Layer 1 (method-registry gate): AIOpsLab/aiopslab/orchestrator/tasks/mitigation.py:66-72
# perform_action() does getattr(self.actions, action_name, None); an unregistered name
# raises InvalidActionError (AIOpsLab/aiopslab/utils/status.py:20-22). Every task type
# (Detection/Localization/Analysis/Mitigation) subclasses the same pattern.
# Layer 2 (content denylist, exec_shell only): AIOpsLab/aiopslab/orchestrator/actions/
# base.py:79-107. exec_shell(command) runs ANY shell command via Shell.exec(command),
# blocked only if it contains one of 5 substrings. exec_shell is inherited by every
# task's Actions class (get_actions() in AIOpsLab/aiopslab/utils/actions.py:51-79 walks
# dir(class_obj), including inherited @action-decorated methods), so it is present in
# every task's available_actions registry, not a rare exception.
from __future__ import annotations
# Reproduced verbatim from AIOpsLab/aiopslab/orchestrator/actions/base.py:93-98.
BLOCK_LIST: dict[str, str] = {
"kubectl edit": "Error: Cannot use `kubectl edit`. Use `kubectl patch` instead.",
"edit svc": "Error: Cannot use `kubectl edit`. Use `kubectl patch` instead.",
"kubectl port-forward": "Error: Cannot use `kubectl port-forward` because it is an interactive command.",
"docker logs -f": "Error: Cannot use `docker logs -f`. Use `docker logs` instead.",
"kubectl logs -f": "Error: Cannot use `kubectl logs -f`. Use `kubectl logs` instead.",
}
# Mitigation task's registered action names: AIOpsLab/aiopslab/orchestrator/actions/
# mitigation.py's MitigationActions subclasses TaskActions and adds only submit();
# get_logs/exec_shell/get_metrics/read_metrics/get_traces/read_traces are inherited.
MITIGATION_ACTIONS = {"submit", "get_logs", "exec_shell", "get_metrics",
"read_metrics", "get_traces", "read_traces"}
class InvalidActionError(Exception):
"""Mirrors AIOpsLab/aiopslab/utils/status.py:20-22 exactly."""
def __init__(self, action_name: str):
super().__init__(f"Invalid action: {action_name}")
self.action_name = action_name
def perform_action(action_name: str, command: str | None = None) -> str:
"""Mirrors MitigationTask.perform_action's getattr-based gate
(AIOpsLab/aiopslab/orchestrator/tasks/mitigation.py:66-72): only names in the
task's registered Actions class may be invoked."""
if action_name not in MITIGATION_ACTIONS:
raise InvalidActionError(action_name)
if action_name == "exec_shell":
return exec_shell(command or "")
return f"executed: {action_name}"
def exec_shell(command: str) -> str:
"""Mirrors AIOpsLab/aiopslab/orchestrator/actions/base.py:79-107 exactly:
a 5-entry SUBSTRING DENYLIST, not an allowlist of safe patterns. Any command
that avoids these 5 substrings runs unmodified."""
for pattern, error in BLOCK_LIST.items():
if pattern in command:
return error
return f"ran: {command}"
# 1. Layer 1 gate: an action name outside the task's registry is rejected, even
# though the caller supplied a well-formed request.
try:
perform_action("delete_namespace")
assert False, "unregistered action name must be rejected"
except InvalidActionError as e:
layer1_reject = str(e)
# 2. Layer 1 gate: a registered, non-exec_shell action succeeds.
layer1_accept = perform_action("get_logs")
assert layer1_accept == "executed: get_logs"
# 3. Layer 2 gate: exec_shell blocks the 5 known-dangerous substrings.
blocked = exec_shell("kubectl edit deploy/checkout-service")
assert blocked.startswith("Error: Cannot use `kubectl edit`")
# 4. Adversarial: exec_shell does NOT block a destructive command outside the
# 5-entry denylist. This is the real gap a "47-pattern whitelist" framing
# would hide: available_actions registers exec_shell as a valid action for
# every task type, and exec_shell's only content filter is the denylist above.
dangerous = exec_shell("kubectl delete namespace kube-system")
assert dangerous == "ran: kubectl delete namespace kube-system"
assert not dangerous.startswith("Error:")
# 5. Adversarial: rm -rf is also not on the denylist.
also_dangerous = exec_shell("rm -rf /var/lib/etcd")
assert also_dangerous == "ran: rm -rf /var/lib/etcd"
print("layer1 reject (unregistered action):", layer1_reject)
print("layer1 accept (registered action):", layer1_accept)
print("layer2 blocked (denylist hit):", blocked)
print("layer2 NOT blocked (outside denylist):", dangerous)
print("layer2 NOT blocked (outside denylist):", also_dangerous)
print("all command-filter assertions passed")
Executed output:
layer1 reject (unregistered action): Invalid action: delete_namespace
layer1 accept (registered action): executed: get_logs
layer2 blocked (denylist hit): Error: Cannot use `kubectl edit`. Use `kubectl patch` instead.
layer2 NOT blocked (outside denylist): ran: kubectl delete namespace kube-system
layer2 NOT blocked (outside denylist): ran: rm -rf /var/lib/etcd
all command-filter assertions passed
The practical conclusion: a production deployment of this architecture must not rely on available_actions containing exec_shell as if that were a safety boundary. If exec_shell is in scope for your deployment's Executor, add a real command allowlist or a much larger denylist in front of it; the shipped 5-entry denylist stops four specific kubectl/docker footguns and nothing else, by design (AIOpsLab is a benchmark harness for a disposable kind cluster, not a hardened production gate).
A real allowlist-based tightening of the exec_shell gate¶
The 5-entry denylist above was checked adversarially beyond the two commands cited in the paper's threat model: it does not stop in-process secret reads (env, printenv, cat /proc/self/environ, all of which would dump OPENROUTER_API_KEY and any other environment-injected secret straight back to the caller) or command injection that rides a ;/&& past the intended single command (e.g. a legitimate-looking kubectl scale ... with && cat /proc/self/environ appended). A fail-closed, argv-level allowlist closes this: reject any command containing a shell metacharacter outright (so injection cannot even reach the tokenizer), then require an exact match against one of the three templates that mirror the Executor's real available_actions shape documented in the RBAC manifest above (kubectl rollout restart, kubectl scale, kubectl delete pod, scoped to hotel-res). Everything else, including every case that bypassed the denylist, is denied by default rather than pattern-matched against a blocklist that can never be exhaustive:
# aoi_allowlist_gate.py - adversarial validation of a fail-closed, argv-level
# allowlist gate for AIOpsLab's exec_shell action, replacing the 5-entry
# substring denylist reproduced and shown bypassable above.
from __future__ import annotations
import re
import shlex
# Part A: the REAL AIOpsLab denylist (verbatim from base.py:93-98).
BLOCK_LIST: dict[str, str] = {
"kubectl edit": "Error: Cannot use `kubectl edit`. Use `kubectl patch` instead.",
"edit svc": "Error: Cannot use `kubectl edit`. Use `kubectl patch` instead.",
"kubectl port-forward": "Error: Cannot use `kubectl port-forward` because it is an interactive command.",
"docker logs -f": "Error: Cannot use `docker logs -f`. Use `docker logs` instead.",
"kubectl logs -f": "Error: Cannot use `kubectl logs -f`. Use `kubectl logs` instead.",
}
def exec_shell_denylist(command: str) -> str:
for pattern, error in BLOCK_LIST.items():
if pattern in command:
return error
return f"ran: {command}"
SECRET_READ_ATTEMPTS = ["env", "printenv", "cat /proc/self/environ", "echo $OPENROUTER_API_KEY"]
INJECTION_ATTEMPTS = [
"kubectl get pods -n hotel-res; env",
"kubectl scale deployment/hotel-res-frontend --replicas=3 -n hotel-res && cat /proc/self/environ",
]
DESTRUCTIVE_OUTSIDE_DENYLIST = ["kubectl delete namespace kube-system", "rm -rf /var/lib/etcd"]
# Part B: argv-level allowlist gate.
_METACHAR_RE = re.compile(r"[;&|`$(){}<>\n]")
ALLOWLIST_PATTERNS = [
re.compile(r"^kubectl rollout restart deployment/[\w.-]+ -n hotel-res$"),
re.compile(r"^kubectl scale deployment/[\w.-]+ --replicas=\d+ -n hotel-res$"),
re.compile(r"^kubectl delete pod [\w.-]+ -n hotel-res$"),
]
class ExecShellDenied(Exception):
pass
def exec_shell_allowlist(command: str) -> str:
"""Fail-closed: reject on any shell metacharacter, then require an exact
match against one of the three real Executor action templates."""
if _METACHAR_RE.search(command):
raise ExecShellDenied(f"rejected: shell metacharacter in {command!r}")
try:
tokens = shlex.split(command)
except ValueError as e:
raise ExecShellDenied(f"rejected: unparseable command {command!r} ({e})")
normalized = " ".join(tokens)
if not any(p.match(normalized) for p in ALLOWLIST_PATTERNS):
raise ExecShellDenied(f"rejected: {command!r} matches no allowlisted Executor action")
return f"ran: {normalized}"
all_bypassed = all(not exec_shell_denylist(c).startswith("Error:")
for c in SECRET_READ_ATTEMPTS + INJECTION_ATTEMPTS + DESTRUCTIVE_OUTSIDE_DENYLIST)
assert all_bypassed, "expected every adversarial case to bypass the 5-entry denylist"
all_now_blocked = True
for cmd in SECRET_READ_ATTEMPTS + INJECTION_ATTEMPTS + DESTRUCTIVE_OUTSIDE_DENYLIST:
try:
exec_shell_allowlist(cmd)
all_now_blocked = False
except ExecShellDenied:
pass
assert all_now_blocked, "every previously-bypassed command must now be denied"
LEGITIMATE = [
"kubectl rollout restart deployment/hotel-res-frontend -n hotel-res",
"kubectl scale deployment/hotel-res-frontend --replicas=3 -n hotel-res",
"kubectl delete pod hotel-res-frontend-abc123 -n hotel-res",
]
for cmd in LEGITIMATE:
assert exec_shell_allowlist(cmd) == f"ran: {cmd}"
print("denylist bypassed by all", len(SECRET_READ_ATTEMPTS + INJECTION_ATTEMPTS + DESTRUCTIVE_OUTSIDE_DENYLIST), "adversarial commands")
print("allowlist now blocks all of them:", all_now_blocked)
print("allowlist still accepts all", len(LEGITIMATE), "legitimate Executor actions")
print("all denylist-bypass and allowlist-gate assertions passed")
Executed output:
denylist bypassed by all 8 adversarial commands
allowlist now blocks all of them: True
allowlist still accepts all 3 legitimate Executor actions
all denylist-bypass and allowlist-gate assertions passed
Each of the 8 adversarial commands was checked individually against both gates, not just summarized: env, printenv, cat /proc/self/environ, and echo $OPENROUTER_API_KEY all ran unmodified through the denylist (ran: env, etc.) and were all denied by the allowlist, the first two on "matches no allowlisted Executor action", the last two on "shell metacharacter in ..." since $ and the injected ;/&& are rejected before the allowlist patterns are even tried; the two destructive commands outside the denylist (kubectl delete namespace kube-system, rm -rf /var/lib/etcd) also ran unmodified through the denylist and were denied by the allowlist for matching no template. This closes the specific gap the audit found (in-process secrets readable, commands outside the small deny list executable) with a fail-closed check rather than a longer denylist that would still be enumerating footguns instead of enumerating what is actually allowed.
Executed: GRPO advantage, the six-dimension reward, and best@k/avg@k¶
This models Eq. 1's group-normalized advantage, Eq. 3's weighted six-dimension step reward with its rule-based hard penalty, and Section 5.1.2's best@k / avg@k metrics. The best@k reconstruction honors Table 9's task-stability bucket counts (14/16/27/29 of 86 tasks); the paper reports only those bucket totals, not which round each task succeeded in, so the per-round assignment below is this page's construction and the resulting curve is a check of the metric definitions, not a reproduction of Figure 7's exact points (best@5 lands at 66.3% by construction, since that value is fixed by the bucket counts alone: (86-29)/86; the intermediate points depend on the random seed).
# aoi_grpo_reward.py - validated: AOI's Observer GRPO group-normalized
# advantage (Eq. 1), the six-dimension weighted step reward with its
# rule-based hard penalty (Sec 3.3.2), and the best@k / avg@k multi-run
# metrics (Sec 5.1.2) applied to a population honoring the paper's Table 9
# task-stability buckets (14/16/27/29 of 86 tasks). Models the mechanisms;
# does not reproduce Qwen3-14B numbers.
from __future__ import annotations
import numpy as np
# Sec 3.3.2 / Appendix A.2: six reward dimensions and default weights.
WEIGHTS = {"format": 0.10, "summary": 0.15, "action": 0.10,
"context_instruction": 0.30, "context_namespace": 0.30,
"confidence": 0.05}
assert abs(sum(WEIGHTS.values()) - 1.0) < 1e-9
assert WEIGHTS["context_instruction"] + WEIGHTS["context_namespace"] == 0.60 # paper's "60%" claim
def step_reward(scores: dict[str, float], valid_json: bool) -> float:
"""Eq. 3: R(x,y) = sum_d w_d * s_d/10. Format is rule-based: a JSON
parse failure triggers a hard penalty R=0.09 regardless of the other
five scores (Sec 3.3.2)."""
if not valid_json:
return 0.09
return sum(WEIGHTS[d] * scores[d] / 10.0 for d in WEIGHTS)
def group_advantage(rewards: np.ndarray, eps: float = 1e-6) -> np.ndarray:
"""Eq. 1: A_i = (R(x,yi) - mu_G) / (sigma_G + eps)."""
mu = rewards.mean()
sigma = rewards.std()
return (rewards - mu) / (sigma + eps)
# 1. A candidate with strong context scores beats one that nails Format and
# Action but gives vague reasoning: Context Instruction + Context
# Namespace alone are 60% of the weight.
strong = {"format": 10, "summary": 8, "action": 10,
"context_instruction": 9, "context_namespace": 9, "confidence": 7}
shallow = {"format": 10, "summary": 5, "action": 10,
"context_instruction": 3, "context_namespace": 3, "confidence": 8}
r_strong, r_shallow = step_reward(strong, True), step_reward(shallow, True)
assert r_strong > r_shallow
# 2. Adversarial: perfect content scores but malformed JSON is capped at the
# hard penalty, below every well-formed candidate regardless of content.
perfect_but_broken = step_reward(
{"format": 10, "summary": 10, "action": 10,
"context_instruction": 10, "context_namespace": 10, "confidence": 10},
valid_json=False)
assert perfect_but_broken == 0.09
assert perfect_but_broken < r_shallow
# 3. GRPO group of G=4 (Table 5): advantages are exactly zero-mean, and the
# highest-reward candidate gets the largest positive advantage.
G = 4
group_rewards = np.array([r_shallow, r_strong, 0.09, step_reward(strong, True) - 0.05])
adv = group_advantage(group_rewards)
assert abs(adv.mean()) < 1e-9
assert int(np.argmax(adv)) == int(np.argmax(group_rewards))
# 4. Adversarial: a degenerate group where every candidate scores identically
# (sigma_G = 0) must not produce NaN/inf; the eps term in Eq. 1 exists
# exactly for this case.
degenerate = np.array([0.5, 0.5, 0.5, 0.5])
adv_degenerate = group_advantage(degenerate)
assert np.all(adv_degenerate == 0.0) and np.all(np.isfinite(adv_degenerate))
# 5. best@k / avg@k (Sec 5.1.2) over a population honoring Table 9's task-
# stability buckets: 14 tasks 5/5, 16 tasks 3-4/5 (split 8/8), 27 tasks
# 1-2/5 (split 14/13), 29 tasks 0/5, 86 tasks total. Per-round pass/fail
# assignment within a task's success count is our reconstruction (the
# paper reports only bucket totals); the metric functions are exact.
rng = np.random.default_rng(0)
n_tasks, n_rounds = 86, 5
counts = np.array([5] * 14 + [4] * 8 + [3] * 8 + [2] * 14 + [1] * 13 + [0] * 29)
assert len(counts) == n_tasks
runs = np.zeros((n_tasks, n_rounds), dtype=bool)
for t, k in enumerate(counts):
idx = rng.choice(n_rounds, size=k, replace=False)
runs[t, idx] = True
def best_at_k(runs: np.ndarray, k: int) -> float:
return float(runs[:, :k].any(axis=1).mean() * 100)
def avg_at_k(runs: np.ndarray, k: int) -> float:
return float(runs[:, :k].mean() * 100)
best_curve = [best_at_k(runs, k) for k in range(1, 6)]
avg5 = avg_at_k(runs, 5)
# Structural properties any valid best@k/avg@k curve must obey, independent
# of our synthetic per-round assignment:
assert best_curve == sorted(best_curve) # best@k is monotone non-decreasing
assert best_curve[-1] <= 100.0 and best_curve[0] >= 0.0
assert avg5 <= best_curve[-1] # avg@5 never exceeds best@5
# A 0/5 task can never contribute to best@k for any k; a 5/5 task always does.
assert runs[counts == 0].any(axis=1).sum() == 0
assert runs[counts == 5].all(axis=1).sum() == 14
print(f"reward: strong={r_strong:.3f} shallow={r_shallow:.3f} broken={perfect_but_broken:.3f}")
print(f"group advantages (G={G}): {np.round(adv, 3).tolist()}, mean={adv.mean():.6f}")
print(f"degenerate-group advantages: {adv_degenerate.tolist()}")
print(f"reconstructed best@k 1..5: {[round(b,1) for b in best_curve]}, avg@5={avg5:.1f}")
print("all GRPO reward and best@k/avg@k assertions passed")
Executed output:
reward: strong=0.895 shallow=0.495 broken=0.090
group advantages (G=4): [-0.267, 0.972, -1.522, 0.817], mean=0.000000
degenerate-group advantages: [0.0, 0.0, 0.0, 0.0]
reconstructed best@k 1..5: [39.5, 54.7, 59.3, 62.8, 66.3], avg@5=38.8
all GRPO reward and best@k/avg@k assertions passed
best@5 (66.3) matches Figure 7's reported point exactly, as it must by construction; best@1 through best@4 in this run (39.5/54.7/59.3/62.8) land close to but not identical with the paper's own curve (31.4/51.2/58.1/62.8), because those intermediate points depend on which of the 5 rounds each partially-successful task landed in, information the paper does not publish per task.
Failure modes¶
| Failure mode | Cause | Fix |
|---|---|---|
| Localization accuracy drops after GRPO training | The reward optimizes end-task completion; the trained Observer explores ~9 more steps per task on average, which surfaces multiple anomaly candidates and picks the wrong one for precision-critical tasks.9 | Track success by task type, not a blended average; consider a task-type-conditioned reward or a separate exploration budget for Localization. |
| Mitigation accuracy stays flat despite Observer training | Remediation commands are generated and executed by the Executor, not the Observer; Observer GRPO cannot directly improve command quality.7 | Train or extend the Executor's command-generation policy separately if Mitigation is the bottleneck; do not expect Observer-only training to move it. |
| A GRPO group produces NaN or divide-by-zero advantages | All G candidates score identically in one training step (sigma_G = 0). |
Eq. 1's + eps term is required, not optional; the executed block's degenerate-group case demonstrates it holds advantages at exactly zero rather than blowing up. |
| Well-formed but shallow diagnostic reasoning scores competitively | A reward implementation that under-weights Context Instruction / Context Namespace against Format/Action lets superficially correct outputs win GRPO groups. | Keep those two dimensions at their paper-default 60% combined weight, and keep Format's hard penalty rule-based and separate from LLM-judged content scores. |
| Benchmark scores read as production-ready | Every number in this paper is an AIOpsLab run; the paper's own Limitations section states production deployment is still future work.10 | Do not cite Table 2-4 numbers as production evidence; if you need a validated production deployment in this KB, see OpsAgent's Lenovo results instead. |
| Confused with the other "AOI" paper in this KB | Two unrelated groups independently chose the acronym AOI for different systems (this page's "Autonomous Operations Intelligence" vs arXiv 2512.13956's "AI-Oriented Operations"). | Always cite the arXiv ID alongside the acronym; link to AOI: AI-Oriented Operations when disambiguation matters. |
| Citing the anonymized review repo as unavailable | The PDF's own Code Availability section points at an anonymous 4open.science link that returns HTTP 401. | Use the de-anonymized project page instead, github.com/OpenEdgeHQ/aoi, verified public via the GitHub API as of 2026-07-16. |
| ~30% of tasks treated as a training target | 29/86 tasks fail across every sampling round, model, and GRPO variant the paper tested; the paper characterizes these as capability gaps, not stochastic noise.9 | Route these fault classes to human escalation rather than budgeting more sampling rounds or retraining against them. |
| Auditing for a "47-pattern Executor whitelist" that does not exist | A repository-wide search for "whitelist"/"allowlist"/"allowed_command" across every .py/.yaml file at commit 17c8f55a0, including the pinned AIOpsLab submodule, returns zero matches; the real constraint is available_actions, AIOpsLab's per-problem API registry passed into ExecutorAgent. |
Audit available_actions for your actual problem/task registration, and add your own command-level filter if you need the guardrail Table 5 describes; do not assume the paper's whitelist artifact ships in this codebase. |
Trusting available_actions as a content-level safety boundary |
exec_shell is registered for every task type (inherited from TaskActions) and runs any shell command via Shell.exec() filtered only by a 5-entry substring denylist (AIOpsLab/aiopslab/orchestrator/actions/base.py:93-98); the executed two-layer command-gate block above shows kubectl delete namespace kube-system, rm -rf /var/lib/etcd, and in-process secret reads (env, cat /proc/self/environ) all pass through unblocked. |
Do not deploy with exec_shell in available_actions against a real estate without adding your own command filter in front of it; the shipped denylist stops four kubectl/docker footguns, nothing else. "A real allowlist-based tightening of the exec_shell gate" above replaces it with a fail-closed, argv-level allowlist, executed and confirmed to block every one of those bypasses while still accepting the three legitimate Executor actions. |
| Reproducing "batch size 16, 3 epochs" from Table 5 | ObserverGRPOConfig's shipped default computes an effective batch size of 32 (batch_size=2 x gradient_accumulation_steps=4 x group_size=4) and num_epochs=5, confirmed by constructing the class directly; only the Evolver's own config (grpo/evolver/grpo_config.py) matches the paper's 3-epoch figure. |
Read the hyperparameters from the config file for the component you are training (Observer vs Evolver), not from Table 5, before citing them as "the AOI setup." |
| Cluster-admin by default | environment/aiopslab_server.py and aiopslab_server2.py both export and set KUBECONFIG from a fresh kind cluster, unconditionally, at module-import time, with no RBAC manifest shipped, the same gap this KB's STRATUS page documents for its AIOpsLab bring-up. |
Fixed here: "The in-cluster credential guard" above patches both files to detect in-cluster execution and build a kubeconfig from the Pod's own ServiceAccount token instead, verified against a real kind cluster with a SelfSubjectAccessReview executed from inside the running Pod (see "Getting the Deployment to Running"). Bind the process to a dedicated, namespace-scoped ServiceAccount/Role/RoleBinding, and apply the same in-cluster guard, before running against anything beyond a disposable test cluster. |
A narrow Executor Role is too narrow for init_problem's own bring-up |
Orchestrator.init_problem() polls pod readiness in the openebs namespace while installing OpenEBS/Prometheus, a wider footprint than the Executor's own available_actions (rollout restart/scale/delete pod, scoped to the target namespace); a real attempt against the RBAC manifest above hit a live 403 Forbidden listing pods in openebs. |
Use a separate, wider bootstrap identity (or a temporarily broader Role) for the one-time environment bring-up phase, and keep the narrow Executor Role above for steady-state diagnosis and mitigation; do not widen the steady-state Role just to make bring-up succeed. |
References¶
- Yang, Chen, Zheng, Li, Li, Tu, Xiao, Pang, Zhang, Li, Long, Ai, Yang, Shi, "AOI: Turning Failed Trajectories into Training Signals for Autonomous Cloud Diagnosis," arXiv:2603.03378: https://arxiv.org/abs/2603.03378
- Code (de-anonymized project page), pinned at commit
17c8f55a030e8850b93c9d16074c2976b6384619(2026-02-11; no LICENSE file at this commit): https://github.com/OpenEdgeHQ/aoi - Published training data and LoRA checkpoints on Hugging Face: seeds
spacezenmasterr/aoi-planner-seeds-sonnet, Observer training dataspacezenmasterr/aoi-observer-training-data, adaptersspacezenmasterr/aoi-evolver-lora-ckpt490andspacezenmasterr/aoi-observer-lora-ckpt200 - AIOpsLab benchmark (Chen et al., MLSys 2025): https://arxiv.org/abs/2501.06706 and https://github.com/microsoft/AIOpsLab, pinned as a submodule of the AOI repository at commit
a56bb5db5d28348dba2ea66ae7693c0b4ee6e6ac(the commit read for "The real Executor gate" and the two-layer command-gate reproduction above) - STRATUS, the multi-agent baseline AOI compares against: https://arxiv.org/abs/2506.02009
- Shao et al., "DeepSeekMath" (GRPO): https://arxiv.org/abs/2402.03300
- Hu et al., "LoRA: Low-Rank Adaptation of Large Language Models": https://arxiv.org/abs/2106.09685
- Kwon et al., "Efficient Memory Management for Large Language Model Serving with PagedAttention" (vLLM): https://arxiv.org/abs/2309.06180
- Yang et al., "Qwen3 Technical Report": https://arxiv.org/abs/2505.09388
- Anthropic, Claude Sonnet 4.5 system card: https://www.anthropic.com/claude-sonnet-4-5-system-card
- Saltzer and Schroeder, "The Protection of Information in Computer Systems" (least-privilege principle the architecture cites): https://doi.org/10.1109/PROC.1975.9939
Related: AIOpsLab · AOI: AI-Oriented Operations · Agentic AIOps · Agentic incident management: OpsAgent · GRPO (group relative policy optimization) · Agent Sandboxing and Isolation
-
AOI (arXiv:2603.03378v3, submitted 2026-03-16, revised 2026-03-17), Gradient / Soochow University / UC Santa Cruz / Georgia Institute of Technology / University College London / Cookiy.ai / WeJoy / ByteDance. Abstract and Section 1 (Introduction): three deployment challenges (proprietary data access, unsafe action execution, closed systems unable to learn from failure); three components (GRPO-trained diagnostic system, read-write separated execution, Failure Trajectory Closed-Loop Evolver); footnote 1 computes the ~215GB deployment cost of a 100B-parameter model under FP16 (~200GB weights + ~10GB KV cache for a 14K-token context + overhead) as the stated reason locally-deployed models must stay under 100B. Purifier strips redundant commands from successful trajectories (Sec 3.3.1). "Code and Data Availability" section: anonymized review repository at anonymous.4open.science/r/aoi-C8C7; "The repository will be made public upon paper acceptance." ↩↩↩↩
-
Section 3 (AOI Runtime Architecture). Table 1: agent roles, permissions, and the Mraw/Mtask/Mcomp access matrix (Observer: Mraw "-", Mtask "R/W", Mcomp "R"; Probe: Mraw "W", Mtask "R", Mcomp "-"; Executor: Mraw "W", Mtask "R", Mcomp "-"; Compressor: Mraw "R", Mtask "-", Mcomp "W"). Section 3.1: three key invariants (Observer cannot read raw outputs; Probe/Executor write raw but cannot read it, all information flows through Compressor; Compressor is stateless per iteration). Section 3.2.1: four-stage pipeline (Decision, Interaction, Compression, Caching). Section 3.2.2: dual-timescale memory, long-term H_{n-2} = {S1,...,S_{n-2}}, short-term C_{n-1} = Compress(RawOutputs_{n-1}). Algorithm 1 (Appendix A.1): main loop with explicit budget-exhaustion return (
E.Submit(timeout)) after N iterations; lazy compression (line 17, after the decision, not before) so the Observer sees fresh evidence when deciding but only compressed summaries persist. ↩↩↩↩↩ -
Section 3.3 (Observer Step-Level Policy Optimization) and Appendix A.2 (Algorithm 2). Eq. 1: group-normalized advantage A_i = (R(x,y_i) - mu_G)/(sigma_G + eps), mu_G = (1/G) sum_j R(x,y_j). Eq. 2: policy gradient over sampled group. Eq. 3 and Section 3.3.2: six reward dimensions and default weights, Format (w=0.10, rule-based, JSON validity; parse failure triggers hard penalty R=0.09), Summary (w=0.15), Action (w=0.10), Context Instruction (w=0.30), Context Namespace (w=0.30), Confidence (w=0.05); Context Instruction + Context Namespace stated as "60% of the total weight." ↩↩
-
Section 4 (Trajectory Evolver). Eq. 4: tau* = pi_evolve(tau, problem); repair for failed seeds, augmentation for success seeds. Section 4.2: seeds defined and categorized (success/failed); Section 4.2: "we use Claude Sonnet 4.5 trajectories on AIOpsLab as a proxy for such expert records... The Evolver's design is agnostic to seed provenance." Eq. 5: same group-normalized advantage form applied to Evolver corrections, scored on Validity/Completeness/Correctness/Effectiveness. Section 4.4 and Appendix A.3/Figure 5: three-stage integration (Failure Collection, Correction Generation, Guidance Injection); corrected plans delivered to the Observer as a "[Corrected Diagnostic Plan]" structured prompt, explicitly guidance not a rigid constraint. ↩↩
-
Section 5.1 (Experimental Setup). Section 5.1.1 and Appendix B.3/Figure 6: nested data split D_obs_train (23 tasks, 11 fault types) subset of D_evolver_train (49 successful Sonnet trajectories) subset of D_all (86 tasks); D_obs_test (63 held-out tasks, 15 unseen fault types plus 15 training-fault-type tasks Sonnet failed); D_evolver_test (37 failed Sonnet trajectories, a strict subset of D_obs_test, so all 37 Evolver test cases fall within the Observer's 63 held-out tasks). Section 5.1.2: best@k (any of k runs succeeds) and avg@k (mean success rate across k runs) defined. Section 5.1.3: Qwen3-14B base, LoRA rank 64/alpha 128/lr 1e-5, GRPO group size G=4, 2xA100 GPUs, vLLM inference. Table 5 (Appendix B.1): max iterations 15, max Probe rounds/iteration 5, context budget 4096 tokens/iteration, long-term memory capacity 10 summaries, Executor whitelist 47 command patterns; GRPO batch size 16, 3 epochs, reward model Claude Opus 4.5, 49 training / 37 test samples for the Evolver. Appendix B.2: AIOpsLab is 88 scenarios reduced to 86 due to 2 deprecated scenarios. ↩↩↩↩↩
-
Table 2 (AOI runtime comparison, full 86-task benchmark, %) and Section 5.2 prose. GPT-4o-mini+AOL-agent: Detection 25.0, Localization 9.5, RCA 7.7, Mitigation 7.7, Overall 14.7. GPT-4o-mini+STRATUS: 78.1/25.0/15.4/23.1/43.0. GPT-4o-mini+AOI: 90.6/32.1/38.5/53.8/58.1 ("Architecture yields 4x improvement over vanilla agents," 58.1/14.7 = 3.95x). Claude Sonnet 4.5+AOL-agent (single-run): 68.8/53.6/15.4/76.9/57.0. Qwen3-14B (best@5/avg@5)+STRATUS: 75.0/41.3, 32.1/11.4, 7.7/4.6, 15.4/15.4, 41.9/22.1. Qwen3-14B (best@5/avg@5)+AOI: 100/66.9, 53.6/27.9, 30.8/7.7, 46.2/23.1, 66.3/38.6. Prose: Detection "100% best@5 vs STRATUS's 75%" (Qwen3-14B best@5 row); RCA "largest relative gains (+150% over STRATUS)" -- this figure arithmetically matches the GPT-4o-mini row's RCA change (15.4->38.5, +150.6%), not the Qwen3-14B best@5 RCA row cited alongside it in the same paragraph (7.7->30.8, +300%), even though the surrounding sentences (Detection, Mitigation) are discussing Qwen3-14B rows -- an inconsistency in the paper's own prose, not introduced by this page; Mitigation "improves 3x over STRATUS" (46.2 vs 15.4, Qwen3-14B best@5); "Qwen3-14B + AOI (66.3% best@5) outperforms Claude Sonnet 4.5 + AOL-agent (57.0%), solving 57 vs 49 tasks" (a 9.3-point gap; distinct from the abstract's "24.4 percentage points" figure, which is 66.3% best@5 minus STRATUS's Qwen3-14B best@5 Overall score of 41.9%, not minus Sonnet's 57.0%). ↩↩↩↩
-
Table 3 (Observer GRPO on held-out fault types, 63 tasks, %) and Section 5.3 prose. Sonnet 4.5 (AOL-agent): Det 54.5, Loc 40.9, RCA 8.3, Mit 57.1, Overall 41.3. AOI (Untrained): 65.5/22.7/6.7/14.3/33.7. AOI (Observer-GRPO): 90.9/18.2/16.7/14.3/42.9. Abstract and Section 1: "lifting avg@1 from 33.7% to 42.9%, surpassing Claude Sonnet 4.5 (41.3%) without multi-run sampling" -- this is the paper's own repeated label for Table 3's numbers. Section 5.3.1: Untrained-to-trained delta (Figure 10) is Detection +36 points (65.5->90.9, prose rounds to "+36.4pp" when compared against Sonnet: 54.5->90.9), Localization -4.5 (22.7->18.2, this is the untrained-vs-trained AOI comparison, not vs Sonnet); "the trained Observer prioritizes high-confidence fault indicators over exhaustive exploration"; Appendix E, ~9 more exploration steps for GRPO-trained vs base. Note: Figure 10's own caption labels these same Detection/Localization deltas "(best@5)," contradicting the abstract's and Section 5.3's "avg@1" label for the identical Table 3 numbers -- a source inconsistency (caption vs. prose) we flag rather than silently resolve; this page follows the abstract/prose "avg@1" label since it is stated more prominently and repeatedly. Sonnet-vs-trained comparison: "improvement concentrates in Detection (+36.4 points) and RCA (+8.4 points)" (54.5->90.9 and 8.3->16.7). Mitigation unchanged 14.3->14.3 because "remediation commands are generated and executed by the Executor, so Observer optimization cannot directly improve the quality of the final repair actions." ↩↩↩↩
-
Table 4 (component ablation on D_evolver_test, 37 tasks, best/avg %; first three rows 5 runs, last row 4 runs) and Sections 5.3.2/5.4. Base: Det 100/50.0, Loc 54/26.2, RCA 27/7.3, Mit 0/0, Overall 54/24.9. Evolver-prompts: 90/64.0, 54/24.6, 18/12.7, 0/0, 49/29.7. Observer-GRPO: 90/64.0, 38/21.5, 36/29.1, 0/0, 49/33.5. Observer-GRPO+Evolver-prompts: 100/72.5, 31/19.2, 36/25.0, 0/0, 49/33.8 ("+8.9 point improvement over Base," 24.9->33.8). Section 5.4.1: Evolver-as-Prompt end-to-end validation (Table 4, avg@5 +4.8%, i.e. Base 24.9 -> Evolver-prompts 29.7) and LLM-judge repair-quality scoring (Claude Opus 4.5) across Validity/Completeness/Correctness/Effectiveness (Figures 3-4). Section 5.4.4: variance analysis, Base best@5-avg@5 gap "29.2pp (54.1%-24.9%)", Evolver-prompts gap "18.9pp (48.6%-29.7%)"; Detection gap 100/50.0 -> 90/64.0 (50pp to 26pp). Section 6 (Discussion) and Section 8 (Conclusion): "GRPO-trained Evolver ... mean LLM judge score 7.18->8.27, std 0.97->0.49"; variance reduction stated as 35% ((29.2-18.9)/29.2 = 35.3%). ↩↩
-
Appendix B.2 (fault types in the 37 failed cases: service failures 31%, misconfigurations 26%, authentication errors 18%, pod failures 15%, network issues 10%; failed-case counts by task type: Detection 10, Localization 13, Mitigation 3, RCA 11). Appendix B.3, Tables 6-7: 11 training fault types / 23 tasks (Table 6); 15 test-only fault types / 48 tasks plus 15 training-fault-type tasks Sonnet failed = 63 held-out (Table 7). Appendix C.1/Figure 7: best@k curve, best@1=31.4%, best@2=51.2% (+19.8pp), best@3=58.1%, best@4=62.8%, best@5=66.3%; k=2 recommended for cost-sensitive deployments, k=3 "captures 88% of the maximum achievable improvement" for high-stakes ones. Appendix C.2, Table 8: per-round success by task type (R1-R5), Overall avg 38.6%; Mitigation flat at 23.1% every round ("failures are capability-limited rather than stochastic"). Appendix C.3, Table 9: task stability distribution over 86 tasks, 5/5 consistently solved 14 (16.3%), 3-4/5 mostly solved 16 (18.6%), 1-2/5 occasionally solved 27 (31.4%), 0/5 never solved 29 (33.7%). Appendix C.4: the 29 never-solved tasks cluster as Localization 13 (astronomy_shop dependency chains), RCA 9 (temporal fault-propagation reasoning), Mitigation 7 (domain-specific remediation, e.g. MongoDB auth recovery, Helm chart upgrades). Appendix D, Figure 11: task-level changes after GRPO training, net +9 tasks (11 improved, 2 degraded, rest unchanged); Appendix D.4: both degraded tasks are Localization (pod_failure_hotel_res-localization-1 and product_catalog_service_failure-localization-1, each 4/5 -> 0/5), root-caused to over-exploration surfacing multiple similar-symptom candidates. Appendix E, Table 10: Observer GRPO training-set composition by task type (Detection 10/43.5%, Localization 6/26.1%, Analysis 1/4.3%, Mitigation 6/26.1%, of 23 total; test set 22/22/12/7 of 63). Table 11: average exploration steps, Observer-GRPO 10.9 vs Base 1.9 (+9.0 average across task types). ↩↩↩↩↩↩↩
-
Section 6 (Discussion): "Safety mechanisms improve capability... Read-write separation forces evidence accumulation before mutation, preventing the cascading failures we observed in STRATUS where premature remediation attempts corrupted system state"; "Capability boundaries are task-specific," 29/86 tasks fail consistently across five Qwen rounds, GPT-4o-mini, and GRPO variants, attributed to systematic gaps (e.g. MongoDB auth recovery needing Helm-specific knowledge) rather than random failure. Section 7 (Limitations): the Evolver "currently generates corrected command sequences as structured prompts"; extending it to "produce synthetic system feedback via environment simulators or to serve as a runtime agent for dynamic plan refinement" is explicitly future work, "beyond the scope of this work." "On the applied side, we plan to deploy AOI in production SRE environments to validate the use and productization potential of the framework within real-world incident response workflows" (i.e., not yet validated in production as of this paper). ↩↩↩↩