Markdown

Earned-verdict pentesting with ptai¶

Scope: ptai (0xSteph/pentest-ai, PyPI ptai), an MIT-licensed AI pentest tool built around one idea: a finding is worthless until a machine oracle re-proves it. This page covers the earned-verdict model (N-of-N idempotent replay plus a false-control twin), the MCP integration that drives it from Claude Code with no API key, the scope-safety guards, and the honest limits of what verification buys. It is the mechanistic counterpart to Shannon's proof-by-exploitation and sits in the agentic cybersecurity and SysOps index.

Examined at commit 96c3bf7f (2026-08-08) of 0xSteph/pentest-ai, MIT, at version 1.1.0. The oracle model below was executed and asserted on this host; separately, the real upstream engine/verifier.py (IdempotentReplayOracle, Verifier) was driven directly with its own intensity table and confirmed the same behavior, including that a non-pass yields candidate, never refuted. No scan was run: that needs a target and an LLM, so the honeypot and Juice Shop numbers quoted are the project's own reproducible benchmarks, not measurements from this host. The project is explicit that these are individual benchmarks, "not field false-positive rates."

What it is¶

ptai is an AI-driven pentest tool that re-runs every exploit to confirm it. It runs recon, logs in, and chains findings into multi-step attack paths, but it refuses to ask you to trust the results. A finding stays a candidate until a machine oracle reproduces it N times out of N; only then does it earn a VERIFIED badge. Third-party scanner output (nuclei, nikto, zap) is held back until an oracle re-proves it independently. Each verified finding ships with a portable proof capsule you can replay yourself (ptai replay). As of 1.1.0 the project reports 14 oracle-verified vulnerability classes.

The enforcement is in code, not policy. engine/verifier.py defines the oracle protocol and the Verifier that turns a candidate into a receipt: the verdict is verified only when the oracle passes, and the receipt must name the oracle that earned it. The two oracles that need no live target are IdempotentReplayOracle (re-runs an attempt N times; verified iff N of N succeed) and DifferentialPrivilegeOracle (the IDOR/BOLA oracle: verified iff an attacker session reads a marker only the victim should). The class-specific oracles in engine/oracles.py add a false-control twin: the same probe is sent with an inert control payload, and if the success marker also appears for the control, the signal is not attributable to the payload, so the oracle abstains.

Why use it¶

Precision you can audit. A VERIFIED verdict names its oracle and carries a replay capsule, so you (or a build gate) can re-prove it. Scanner noise is what trains teams to ignore their tools; ptai reports only what it proved.
No API key on the MCP path. Wire it into Claude Code as an MCP server (claude mcp add pentest-ai -- ptai mcp) and your existing subscription is the LLM; the tools and 60-plus probes run locally against your target.
Honest about itself. The project publishes both a curated Juice Shop benchmark (breadth) and a private honeypot harness with bugs the authors wrote themselves (the honest signal), and states plainly that the oracle gate "buys precision, not catch rate."
Scope-locked active tools. As of 1.1.0, active tools (sqlmap, dalfox) are host-locked to the engagement target, and the scan no longer feeds third-party URLs scraped from page content to attack tools.

When to use it (and when not)¶

Use it when you want low-false-positive findings on a web target you are authorized to test, especially driven interactively from Claude Code, and when a replayable proof of each finding matters (a CI gate that breaks a build only on proven findings, --fail-on verified). It is a good fit for authorized engagements where the cost of a false alarm is high.

Do not read the benchmark numbers as field detection rates: the oracle gate removes false positives, it does not raise catch rate, so ptai detects more than it verifies and only the verified subset reaches the report. Against a novel target, the catch rate is whatever the curated probe library covers, not the Juice Shop figure. This is offensive tooling: installing accepts the acceptable-use policy, and you must have authorization. Do not confuse a candidate verdict with "safe": a replay miss means the tool could not re-prove the finding, not that the vulnerability is absent.

Architecture¶

flowchart TB
  DRIVE["Claude Code over MCP (no API key)"] --> ENGINE["ptai engine: recon, probes, chains"]
  ENGINE --> CAND["Candidate finding"]
  CAND --> ORACLE{"Machine oracle"}
  ORACLE -->|"N of N replay AND control abstains"| VERIFIED["VERIFIED + proof capsule"]
  ORACLE -->|"any miss OR control also fires"| CANDIDATE["Candidate (abstain, not disproof)"]
  VERIFIED --> REPORT["Report: proven findings only"]
  VERIFIED --> REPLAY["ptai replay: re-prove yourself"]

The earned verdict, executed¶

Two properties make the VERIFIED badge mean something. First, N-of-N idempotent replay kills single-shot luck: the exploit must reproduce every time, and N scales with intensity (safe=2, normal=3, aggressive=5, from REPLAY_N_BY_INTENSITY). Second, the false-control twin kills non-attributable signal: if an inert control payload produces the same success marker (for example a server that echoes every input), the oracle cannot attribute the result to the attack and abstains. A non-pass is always an abstain (candidate), never a disproof; refuted stays reserved for a future real-disprove oracle. The model below reproduces both gates.

REPLAY_N_BY_INTENSITY = {"safe": 2, "normal": 3, "aggressive": 5}


def replay_verdict(attempt, n):
    """attempt() -> bool, run n times. Verified iff every run succeeds."""
    successes = sum(1 for _ in range(n) if attempt())
    passed = successes == n
    return ("verified" if passed else "candidate", successes, n)


def control_twin_verdict(payload_marker, control_marker, attempt, n):
    """Verified iff the payload reproduces N/N AND the inert control never does."""
    verdict, s, _ = replay_verdict(attempt, n)
    if verdict == "verified" and control_marker:
        return "candidate", s, n   # control produced the marker too -> not attributable
    return verdict, s, n


# 1. Stable-true exploit at each intensity -> VERIFIED, with N scaling.
for inten, n in REPLAY_N_BY_INTENSITY.items():
    v, s, a = replay_verdict(lambda: True, n)
    assert v == "verified" and a == n
    print(f"intensity {inten:<10} -> {v} {s}/{a}")

# 2. A flaky exploit (2 of 3) is an ABSTAIN, not a refutation.
seq = iter([True, False, True])
v, s, a = replay_verdict(lambda: next(seq), 3)
assert v == "candidate" and s == 2
print(f"flaky 2/3            -> {v} {s}/{a}  (abstain: could not re-prove, not 'safe')")

# 3. False-control twin: marker present for payload AND for the inert control -> abstain.
v, s, a = control_twin_verdict(payload_marker=True, control_marker=True,
                               attempt=lambda: True, n=3)
assert v == "candidate"
print(f"payload+control both -> {v} {s}/{a}  (marker not attributable to the payload)")

# 4. Marker present for the payload, ABSENT for the control -> earned VERIFIED.
v, s, a = control_twin_verdict(payload_marker=True, control_marker=False,
                               attempt=lambda: True, n=3)
assert v == "verified"
print(f"payload only         -> {v} {s}/{a}  (control abstained; badge is earned)")
print("OK: verification buys precision (kills luck + non-attributable signal), not catch rate")

Executed output:

intensity safe       -> verified 2/2
intensity normal     -> verified 3/3
intensity aggressive -> verified 5/5
flaky 2/3            -> candidate 2/3  (abstain: could not re-prove, not 'safe')
payload+control both -> candidate 3/3  (marker not attributable to the payload)
payload only         -> verified 3/3  (control abstained; badge is earned)
OK: verification buys precision (kills luck + non-attributable signal), not catch rate

Running the real engine/verifier.py gave the same shape: IdempotentReplayOracle.for_intensity(attempt, "aggressive") yields attempts=5, a stable-true attempt returns passed=True, and Verifier.verify_candidate maps a 2-of-3 replay to Verdict.candidate while a 3-of-3 maps to Verdict.verified. The abstain-not-disprove rule is a comment-documented invariant in that file, and it holds in execution.

How to use it¶

Install and try the offline demo, which needs no target and no key:

pip install ptai && ptai demo

ptai demo scans a bundled vulnerable app, reports 4 findings, 4 oracle-VERIFIED, replays one live from a proof capsule, then runs the same routes hardened and reports 0 findings. The findings appear and disappear with the fix, which is the point. To drive real engagements from Claude Code:

claude mcp add pentest-ai -- ptai mcp

Then ask Claude Code to run an authorized, authenticated pentest against your staging host. The MCP surface exposes tools to list and run 200-plus wrapped security tools, run 60-plus SPA-aware probes, issue raw HTTP under a scope guard, and manage the engagement record.

How to develop with it¶

New coverage is a new oracle. Each oracle in engine/oracles.py returns an OracleResult and, for the control-twin classes, defines an inert control payload that must be a fixed literal rather than a per-run nonce (because the verdict is idempotent N-of-N replay, the control string has to stay stable across every replay). The honeypot harness under tests/honeypot/ and the clean-app zero-false-positive gate under tests/cleanapp/ are where a new oracle earns its keep: the honeypot has a control that must fail on a safe target, so a non-vulnerable app abstains instead of earning a badge. The per-oracle tests (tests/test_oracle_*.py) are the template for adding one.

How to maintain it¶

Pin the PyPI version; verification coverage roughly doubled from the pre-1.1.0 release to 14 classes, and the semantics of a scan's zero-result changed (a fragile single-container target that fell over mid-scan used to report 0; 1.1.0 waits for the target to answer again before re-proving, which took a Juice Shop scan from 0 to 12 verified). Re-read engine/oracles.py on update to see which classes gained oracles. Keep the proof capsules from past runs; they are the durable artifact you replay to confirm a finding still reproduces after a fix.

How to run it in production¶

Production use is a CI security gate on authorized targets. The --fail-on verified flag breaks a build only on proven findings, which is the safe default: a candidate never fails the build, so scanner noise does not block delivery. Keep active tools host-locked (the 1.1.0 default) so an aggressive sweep cannot wander to a scraped third-party URL. Run the engine's local tools behind the same sandbox and egress controls you apply to any agent that executes model-chosen commands, and keep the MCP path's prompt-and-output flow in mind: on the MCP path your prompts and the tool output Claude Code reads go through the provider's API like any Claude Code session, so use the Ollama or on-prem path if you need an air-gapped run.

Failure modes¶

Verification is not detection. The oracle gate removes false positives; it does not find anything the probes missed. System recall is bounded by the probe library, which is curated and growing, not exhaustive.
Benchmark reading. Juice Shop is the most-documented vulnerable app on the internet, so its verified count reads as a precision story, not a field catch rate. The honeypot numbers are lower on purpose and are the honest signal.
Candidate is not safe. A replay miss or a control that also fires means "could not re-prove," not "not vulnerable." No oracle here proves a target safe.
Aggressive intensity can knock a target over. More replays and probes stress a fragile single-container target; the verify phase now waits for recovery, but a truly fragile target still needs care.
Offensive tooling. Authorization and the acceptable-use policy are prerequisites, not formalities.

References¶

ptai / pentest-ai repository (0xSteph), pinned commit 96c3bf7f: https://github.com/0xSteph/pentest-ai
Verifier and oracle protocol (engine/verifier.py): https://github.com/0xSteph/pentest-ai/blob/main/engine/verifier.py
Class oracles (engine/oracles.py): https://github.com/0xSteph/pentest-ai/blob/main/engine/oracles.py
Why verification: https://github.com/0xSteph/pentest-ai/blob/main/docs/why-verification.md
Juice Shop benchmark: https://github.com/0xSteph/pentest-ai/blob/main/docs/benchmarks/juice-shop.md
PyPI: https://pypi.org/project/ptai/