Markdown

AI SRE agents with OpenSRE¶

Scope: OpenSRE (Tracer-Cloud/opensre), an Apache-2.0 framework for building AI Site Reliability Engineering agents that investigate and resolve production incidents on your own infrastructure. This page covers the tool-calling investigation loop, the reversible identifier masking that lets teams use external models without leaking pod and cluster names, the 60-plus integrations, and the synthetic-incident environment the project uses as a training and evaluation ground. It is the operations counterpart to the security tools in the agentic cybersecurity and SysOps index, and it sits next to agentic AIOps, agentic incident management, and STRATUS.

Examined at commit dceb3c3d (2026-08-09) of Tracer-Cloud/opensre, Apache-2.0, described by the project as public alpha ("core workflows are usable for early exploration, though not yet fully stable"). The masking model below was executed and asserted on this host; separately, the real upstream platform/masking modules (context.py, detectors.py) were copied out and driven directly, and they round-tripped a realistic Datadog k8s alert byte-for-byte, including the false-positive shown below. No live investigation was run: that needs cloud credentials and an LLM key, so every capability claim here is read from the code, not a benchmark. The repo ships no benchmark numbers ("No benchmark results yet." under make benchmark).

What it is¶

OpenSRE is a framework, not a single agent. Its core (core/agent_harness) is a tool-calling loop: an alert arrives, the agent fetches correlated context (logs, metrics, traces, recent deploys), reasons across your connected systems to test hypotheses, and produces a structured investigation report with a probable root cause and linked evidence. It can then suggest or, optionally, execute remediation, and post a summary to Slack, PagerDuty, or Telegram. You drive it three ways: an interactive REPL (opensre), a one-shot investigation against an alert file (opensre investigate -i alert.json), or from Python via AgentSession.

The project's framing is deliberately research-shaped. Its thesis is that coding agents got good because SWE-bench gave them scalable training data and clear feedback, and that production incident response still lacks that. So OpenSRE ships not only the agents but the environment they need to improve: scored synthetic root-cause suites under tests/synthetic (which check root-cause accuracy, required evidence, and planted adversarial red herrings) and real end-to-end scenarios under tests/e2e across Kubernetes, EC2, CloudWatch, Lambda, ECS Fargate, and Flink.

The load-bearing safety feature is reversible identifier masking (platform/masking). Before evidence goes to an external LLM, sensitive infrastructure identifiers (pod, namespace, cluster, hostname, account id, IP, email, service name) are replaced with stable placeholders like <POD_0> and <NAMESPACE_1>; the model reasons over masked text; the originals are restored in any user-facing output. It is off by default and enabled per investigation through environment variables.

Why use it¶

Own-infrastructure incident response. The agent runs where your telemetry already lives and reads it through 60-plus integrations rather than expecting you to export everything to a vendor.
Evidence-backed conclusions. Every root cause links to the data behind it, which is the difference between a usable RCA and a plausible guess.
Bring your own model. Anthropic, OpenAI, Codex, Ollama, Gemini, OpenRouter, NVIDIA NIM, and Bedrock are all wired in, so you choose the cost and privacy posture.
A training ground, not just a runtime. The synthetic-incident suites make the agent's quality measurable and improvable, which is rare in this class of tool.
Reversible masking as defense-in-depth. You can use a frontier hosted model while keeping raw pod and cluster names out of the request.

When to use it (and when not)¶

Use it when you have real telemetry across several systems and want an agent to do the first-pass correlation an on-call engineer does at 3am, and when you can run it on infrastructure you control. It fits teams that want to shape the agent to their runbooks and integrations rather than accept a closed product.

Do not treat it as production-hardened: it is public alpha, APIs and integrations move, and it ships no benchmark numbers, so measure it on your own incidents before trusting a remediation path. Do not rely on masking as a privacy guarantee (see Failure modes): the detectors are heuristic and will both miss real identifiers and over-capture ordinary prose. Do not let it execute remediation unattended without the risk-tiered approval gates this knowledge base recommends for any state-mutating agent.

Architecture¶

flowchart TB
  ALERT["Alert or query"] --> HARNESS["Agent harness: tool-calling loop"]
  subgraph TOOLS["Connected systems (60+)"]
    OBS["Observability: Grafana, Datadog, CloudWatch"]
    INFRA["Infra: Kubernetes, AWS, GCP, Azure"]
    DEPLOY["Deploys and config"]
  end
  HARNESS -->|"fetch context"| TOOLS
  TOOLS --> MASK["Masking: identifiers to placeholders"]
  MASK --> LLM["External LLM (sees masked evidence)"]
  LLM --> UNMASK["Unmask for user-facing output"]
  UNMASK --> REPORT["RCA report: root cause plus linked evidence"]
  REPORT --> SINK["Slack, PagerDuty, Telegram"]

How masking works, executed¶

The masking round trip is the piece worth understanding in detail, because it is what makes an external model usable on private telemetry. Placeholders are per-kind and stable (the same value always maps to the same token within one investigation), the map is serialized into investigation state so it survives node-to-node handoffs, and unmask is a single left-to-right scan over a <[^<>]+> token regex so a longer placeholder is never partially rewritten by a shorter key. The model below reproduces that mechanism faithfully, including the counter-resume across a state round trip and the contextual-detector false positive.

import re

# Contextual detectors capture group(1) = the value after a label keyword.
NS = re.compile(r"\b(?:kube_namespace|namespace|ns)[=:\s]+([a-z0-9][-a-z0-9]*)\b", re.I)
POD = re.compile(r"\b([a-z0-9](?:[-a-z0-9]*[a-z0-9])?-[a-f0-9]{5,10}(?:-[a-z0-9]{3,10})?)\b")
DETECTORS = [("NAMESPACE", NS), ("POD", POD)]
TOKEN = re.compile(r"<[^<>]+>")  # single-bracket invariant: one scan finds every token


class Masker:
    def __init__(self, placeholder_map=None):
        self._map = dict(placeholder_map or {})            # placeholder -> original
        self._rev = {v: k for k, v in self._map.items()}   # original -> placeholder
        maxi = {}                                          # resume counters per kind
        for ph in self._map:
            kind, _, idx = ph.strip("<>").rpartition("_")
            if kind and idx.isdigit():
                maxi[kind] = max(maxi.get(kind, -1), int(idx))
        self._counters = {k: v + 1 for k, v in maxi.items()}

    def _placeholder(self, kind, value):
        if value in self._rev:
            return self._rev[value]                        # stable: same value, same token
        i = self._counters.get(kind, 0)
        self._counters[kind] = i + 1
        ph = f"<{kind}_{i}>"
        self._map[ph] = value
        self._rev[value] = ph
        return ph

    def mask(self, text):
        found = []
        for kind, rx in DETECTORS:
            for m in rx.finditer(text):
                s, e = m.span(1) if m.groups() else m.span()
                found.append((s, e, kind, m.group(1) if m.groups() else m.group()))
        found.sort(key=lambda t: (t[0], -(t[1] - t[0])))   # longest match wins at each start
        parts, cursor, last_end = [], 0, -1
        for s, e, kind, val in found:
            if s < last_end:
                continue
            parts.append(text[cursor:s])
            parts.append(self._placeholder(kind, val))
            cursor, last_end = e, e
        parts.append(text[cursor:])
        return "".join(parts)

    def unmask(self, text):
        if "<" not in text:
            return text
        return TOKEN.sub(lambda m: self._map.get(m.group(0), m.group(0)), text)

    def state(self):
        return dict(self._map)


alert = "pod api-7d9f8b-xkp2q in kube_namespace:tracer-test scaled to 0"
m = Masker()
masked = m.mask(alert)
assert m.unmask(masked) == alert                           # 1. round trip is lossless
print("masked:", masked)

two = Masker()
out = two.mask("ns:app-a and ns:app-a and ns:app-b")       # 2. stable placeholders
assert out.count("<NAMESPACE_0>") == 2 and "<NAMESPACE_1>" in out
print("stable :", out)

big = Masker()
txt = " ".join(f"ns:x-{i}" for i in range(12))             # 3. 10+ tokens, no clobber
mtxt = big.mask(txt)
assert big.unmask(mtxt) == txt
print("ns#11  :", mtxt.split()[-1])

resumed = Masker(placeholder_map=big.state())              # 4. counters resume across state
nxt = resumed.mask("ns:x-new")
assert "<NAMESPACE_12>" in nxt and resumed.state()["<NAMESPACE_0>"] == "x-0"
print("resume :", nxt, "(old <NAMESPACE_0> still intact)")

leak = Masker().mask("the ns was scaled down")             # 5. THE FALSE POSITIVE
print("f-pos  :", leak, "<- 'was' captured as a namespace value")
assert "<NAMESPACE_0>" in leak
print("OK: round-trip is exact; the detector is heuristic, so masking is defence-in-depth")

Executed output:

masked: pod <POD_0> in kube_namespace:<NAMESPACE_0> scaled to 0
stable : ns:<NAMESPACE_0> and ns:<NAMESPACE_0> and ns:<NAMESPACE_1>
ns#11  : ns:<NAMESPACE_11>
resume : ns:<NAMESPACE_12> (old <NAMESPACE_0> still intact)
f-pos  : the ns <NAMESPACE_0> scaled down <- 'was' captured as a namespace value
OK: round-trip is exact; the detector is heuristic, so masking is defence-in-depth

The false positive is not a nitpick. The ns keyword detector captures the next word after ns, so ordinary prose ("the ns was scaled down") sends was to the placeholder map. Harmless here, but it means masking changes the text the model sees in ways that are hard to predict, and it is why OpenSRE pairs masking (reversible, for identifiers) with a separate one-way GuardrailEngine (irreversible [REDACTED], for hard-block secrets like API keys and credit cards). Masking is defense-in-depth, not a boundary you can prove.

How to use it¶

Install with the shell one-liner (curl -fsSL https://install.opensre.com | bash) or brew install tracer-cloud/tap/opensre, then opensre onboard. Run an investigation against a captured alert:

opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json

Turn masking on for that run:

export OPENSRE_MASK_ENABLED=true
export OPENSRE_MASK_KINDS=pod,namespace,cluster,hostname
opensre investigate -i tests/e2e/kubernetes/fixtures/datadog_k8s_alert.json

OPENSRE_MASK_EXTRA_REGEX accepts a JSON object mapping a label to a custom regex (group 1 is the span to mask) for identifiers the built-ins miss, such as a Jira key. Policies are re-read from the environment at the start of each investigation, so changes take effect on the next run without a restart.

How to develop with it¶

The Python API drives the agent in-process:

# Reference template, unexecuted here (needs the source checkout and an LLM key).
from core.agent_harness import AgentSession

session = AgentSession.start()
result = session.chat("why is checkout-api slow?")
if result.answered:
    print(result.primary_response_text)

Development uses uv: make install then uv run opensre .... The repo's own AGENTS.md is unusually specific about conventions (named http.HTTPStatus constants over literals, shared env names under config/constants/, docstring-only Protocol bodies), so match them when contributing. The synthetic suites under tests/synthetic are the place to add a new scored incident: each mock backend (mock_aws_backend, mock_datadog_backend, mock_grafana_backend) plus a scenario test defines a failure with a known root cause and planted red herrings.

How to maintain it¶

Regenerate benchmark numbers with make benchmark and refresh the README table via make benchmark-update-readme. Telemetry (PostHog product analytics, Sentry errors) is opt-out; disable with export OPENSRE_NO_TELEMETRY=1. Because the project is public alpha and moves quickly, pin to a commit for any deployment and re-diff platform/masking and the integration catalog on update: the masking detectors and the 60-plus integration list are the two surfaces most likely to change under you.

How to run it in production¶

Two AWS EC2 paths and a hosted option are documented. The gateway path (make build-gateway-image, make deploy-gateway) installs a Telegram gateway into a server image with no Docker. The hosted path deploys the repo Dockerfile to Railway, ECS, or Vercel; set LLM_PROVIDER and the matching API key, plus DATABASE_URI and REDIS_URI if you need persistence. In production, keep masking on for any external model, keep remediation behind human approval until you trust it on your own incident history, and treat the agent's egress like any other workload that reaches your cloud APIs: put it behind the runtime enforcement layer and the sandbox.

Failure modes¶

Masking misses. The detectors are keyword-and-shape heuristics. A hostname that does not match the DNS-style pattern, a bare pod name without a hex suffix, or a bearer token all pass straight through to the model. Never present masking as a privacy boundary; it lowers exposure, it does not eliminate it.
Masking over-captures. As shown, ordinary words after a bare ns or service keyword get masked, which perturbs the evidence text the model reasons over. Prefer the more specific label forms (kube_namespace:) in your telemetry where you can.
Public alpha drift. APIs, integrations, and the masking module change; a pinned deployment can silently diverge from main.
No published benchmarks. The project ships the environment to measure quality but no numbers, so the agent's root-cause accuracy on your stack is unknown until you run the synthetic and e2e suites yourself.
Remediation is real. An agent that can execute actions against your cloud can make an incident worse. Gate it.

References¶

OpenSRE repository (Tracer-Cloud), pinned commit dceb3c3d: https://github.com/Tracer-Cloud/opensre
Masking design doc: https://github.com/Tracer-Cloud/opensre/blob/main/docs/masking.mdx
Masking source (platform/masking/context.py, detectors.py, policy.py): https://github.com/Tracer-Cloud/opensre/tree/main/platform/masking
Synthetic incident suites: https://github.com/Tracer-Cloud/opensre/tree/main/tests/synthetic
Product docs: https://www.opensre.com/docs
SWE-bench (the motivating analogy): https://arxiv.org/abs/2310.06770