Copy for agents Markdown

Agentic vulnerability scanning at repository scale

Scope: running a coding agent as a security reviewer over an entire existing codebase, rather than over a diff. This page covers the funnel architecture that keeps the cost bounded, the on-disk model that makes multi-worker and resumable scans possible, how false positives are suppressed, and the isolation posture the scanner itself needs. The defensive counterpart, defending agents from attackers, is in the agent threat model; the offensive-capability framing is in offensive AI security.

The system described is vercel-labs/deepsec, Apache-2.0, examined at commit 76c03d67b1a47882c9e4bb738b57253155307945 (authored 2026-07-26). The matcher audit below was executed against that clone on this host with Node v25.8.2 using native TypeScript type stripping; no pnpm install was performed, so 118 of the 200 matcher modules could not be loaded and are excluded from the counts. No scan was run: that requires paid model credentials, so every cost and detection figure in the Python model is this page's own parameterized model, not a measurement of deepsec.

flowchart LR
  REPO["Repository (all files)"] --> SCAN["scan: 200 regex matchers<br/>free, no AI"]
  SCAN --> CAND["Candidate sites only"]
  CAND --> PROC["process: coding agent<br/>investigates each batch"]
  PROC --> FIND["Findings + recommendation"]
  FIND --> REVAL["revalidate: re-read code<br/>+ check git history"]
  REVAL --> VERDICT["true-positive / false-positive<br/>fixed / uncertain"]
  VERDICT --> EXPORT["export / report / metrics"]

What it is

An agentic scanner inverts the usual static-analysis tradeoff. Classic SAST is cheap per file and drowns reviewers in low-confidence findings; a coding agent reasons well about whether something is genuinely exploitable but costs real money per file. Running the agent over an entire monorepo is financially impossible, and the tool's own README says so directly: scans "can cost thousands or even tens-of-thousands of dollars for large codebases."

The resolution is a two-stage funnel:

1. scan globs the repository and runs a registry of regex matchers over every file, writing candidate sites. This stage uses no AI at all and is documented as taking roughly 15 seconds for 2,000 files.
2. process batches the files that have candidates, sends each batch to a coding-agent backend along with a repository-context document, and parses structured findings back.

Everything after that shapes the result: revalidate re-reads the code and consults git history to mark findings true-positive, false-positive, fixed, or uncertain; triage applies a cheaper model for P0/P1/P2 classification; enrich attaches committer and ownership data; report, export, and metrics are read-only.

Three agent backends are interchangeable behind the same prompt and output schema, which matters operationally because it decouples the pipeline from any single vendor:

--agent	SDK	Default model
codex (default)	@openai/codex-sdk	gpt-5.5
claude	@anthropic-ai/claude-agent-sdk	claude-opus-4-8
pi	@earendil-works/pi-coding-agent	zai/glm-5.2

Backends can be mixed within one project: re-processing a file with a different agent merges the second run's findings with the first.

Why the architecture is shaped this way

The funnel is a cost-control mechanism, and it sets a hard recall ceiling. A file with no candidate match is never seen by the agent. That means matcher recall, not model capability, bounds what the whole system can find. Everything else about the design follows from taking that seriously: the project ships 200 matchers, documents how to write more, and explicitly tells users to prompt a coding agent to grow the matcher set for their codebase.

The on-disk model is append-only, which is what makes resumability and parallelism fall out for free. Each scanned file gets one JSON FileRecord that is the source of truth for everything known about it: candidate matches, findings, analysis history, git committer info, ownership. Stages merge rather than overwrite. A re-scan merges new candidates; a re-process appends to analysisHistory; revalidation tags existing findings with verdicts. Nothing is deleted. An interrupted run is resumed by re-running the same command, which skips files already analyzed.

Work claiming is atomic, so horizontal scale needs no coordinator. The processor claims files via a lockedByRunId field, letting multiple workers run against the same data directory without stepping on each other. Distributed execution over microVMs is then just more workers: --sandboxes 10 --concurrency 4.

Revalidation exists because a single pass is not precise enough. The project documents that revalidation "empirically reduces FP rate by 50%+ on most repos" and costs roughly as much as process, recommending it for high-severity findings rather than universally.

When to use it (and when not)

Use an agentic full-repo scan when:

• The target is a large existing codebase with unreviewed history. The stated design goal is surfacing issues "that have been lurking in applications for a long time." This is a backlog-clearing tool.
• You can attach a dollar budget to security findings. The economics only work if a found vulnerability is worth more than the four-to-five-figure scan.
• You control the infrastructure it runs on. Running in your own environment is the point; source code never has to leave.

Use the diff mode, not the full scan, when:

• You want continuous coverage. process --diff restricts scan and investigation to files changed in a diff, which is the PR-review and CI-gating path. This is the mode to run per-commit; full scans are periodic events.

Do not use it when:

• You need deterministic, reproducible results. Findings come from a model at high reasoning effort; two runs will not agree exactly.
• You expect it to replace matchers with intelligence. It cannot investigate what the regex stage never surfaced.
• You cannot isolate it. See the isolation section: this is a coding agent with shell access.

How to develop with it: the matcher is the leverage point

Because matcher recall caps system recall, writing matchers is the highest-value work. A matcher is a small module declaring a slug, a noise tier, the file globs it applies to, a set of illustrative examples, and a match(content, filePath) function returning candidate sites with line numbers and snippets.

The examples field is not decoration: it is a self-test. Running every matcher against its own declared examples is a cheap, complete audit of the registry, and it is exactly what the harness below does.

// matcher_audit.mjs - run from a directory containing a clone of vercel-labs/deepsec
// Audits deepsec's regex matchers against their own declared examples.
import { readdirSync } from "node:fs";

const DIR = new URL("./deepsec/packages/scanner/src/matchers/", import.meta.url);

// Candidate paths: derived from each matcher's own filePatterns, plus the
// framework-internal shapes some matchers gate on inside match().
function candidatePaths(globs) {
  const out = new Set(["src/server/render.ts", "packages/next/src/server/image.ts"]);
  for (const g of globs ?? []) {
    let p = g.replace(/\*\*\//g, "").replace(/\{([^},]+)[^}]*\}/g, "$1").replace(/\*/g, "route");
    out.add("app/" + p.replace(/^\/+/, ""));
  }
  return [...out];
}

const files = readdirSync(DIR).filter((f) => f.endsWith(".ts")).sort();
let loaded = 0, skipped = 0, total = 0, matched = 0;
const misses = [];
const gated = [];

for (const f of files) {
  let mod;
  try { mod = await import(new URL(f, DIR)); loaded++; }
  catch { skipped++; continue; }               // needs the workspace package at runtime
  for (const m of Object.values(mod)) {
    if (!m?.match || !Array.isArray(m.examples)) continue;
    const paths = candidatePaths(m.filePatterns);
    // Does the matcher gate more narrowly than its declared filePatterns?
    const broad = "app/" + (m.filePatterns?.[0] ?? "").replace(/\*\*\//g, "")
      .replace(/\{([^},]+)[^}]*\}/g, "$1").replace(/\*/g, "route");
    let broadHits = 0;
    for (const [i, ex] of m.examples.entries()) {
      total++;
      if (m.match(ex, broad).length > 0) broadHits++;
      if (paths.some((p) => m.match(ex, p).length > 0)) matched++;
      else misses.push(`${m.slug}[${i}]`);
    }
    if (broadHits === 0 && m.examples.length > 0) gated.push(m.slug);
  }
}

console.log(`matcher files in the registry:        ${files.length}`);
console.log(`loaded with bare node type-stripping: ${loaded} (${skipped} import @deepsec/core at runtime)`);
console.log(`declared examples executed:           ${total}`);
console.log(`examples that match on some declared path: ${matched} (${(100 * matched / total).toFixed(1)}%)`);
console.log(`examples that never match:            ${misses.length}${misses.length ? " -> " + misses.join(", ") : ""}`);
console.log(`matchers gating narrower than filePatterns: ${gated.length} -> ${gated.join(", ")}`);

Executed output (node matcher_audit.mjs, Node v25.8.2):

matcher files in the registry:        200
loaded with bare node type-stripping: 82 (118 import @deepsec/core at runtime)
declared examples executed:           681
examples that match on some declared path: 680 (99.9%)
examples that never match:            1 -> framework-server-action[10]
matchers gating narrower than filePatterns: 1 -> framework-untrusted-fetch

Two results from that run are worth acting on.

filePatterns is a pre-filter, not the applicability contract. The framework-untrusted-fetch matcher declares filePatterns: ["**/*.ts", "**/*.tsx", "**/*.js", "**/*.mjs"] but its match() body immediately returns nothing unless the path looks like framework internals (src/server/, packages/next/src/, and similar). All seven of its examples match on src/server/render.ts and none on app/route.ts. You therefore cannot audit which files a matcher will actually flag by reading its globs; you have to read the body. When writing matchers, keep the real gate in filePatterns where the scanner can see it, or document the internal gate.

One declared example does not match its own matcher, and the cause is a regex defect. framework-server-action example 10 is const id = req.headers.get('next-action-id');. Its content hint is /\b['"]next-action(?:-id|-redirect)?['"]/. The leading \b sits immediately before a character class of quote characters, so it demands a word character immediately preceding the quote, which does not occur in valid JavaScript:

const RE = /\b["']next-action(?:-id|-redirect)?["']/;
RE.test("req.headers.get('next-action-id')");  // false  <- the real-world shape
RE.test("x = 'next-action-id'");               // false
RE.test("['next-action-id']");                 // false
RE.test("a'next-action-id'");                  // true   <- not valid JavaScript
// Dropping the leading \b makes all four true.

Executed output:

false  quoted arg after (     "req.headers.get('next-action-id')"
false  after space            "x = 'next-action-id'"
true   after word char        "a'next-action-id'"
false  after [                "['next-action-id']"
--- without the leading \b ---
true   quoted arg after (     "req.headers.get('next-action-id')"
true   after space            "x = 'next-action-id'"
true   after word char        "a'next-action-id'"
true   after [                "['next-action-id']"

That hint is effectively dead code at the audited commit. The matcher still fires through its other content hints and its path hint, so the practical impact is narrow, but it is a genuine silent recall gap and it illustrates the general hazard: a matcher that never fires costs nothing and reports nothing, so it fails invisibly. Anchoring a word boundary against punctuation is the most common way to write one by accident.

How to size and budget a scan

The funnel's economics are the decision most teams get wrong, in both directions. Run: python3 scan_economics.py.

import numpy as np

# Illustrative repository and price inputs. These are NOT deepsec measurements;
# substitute your own file count, match rate, and per-file agent cost.
FILES = 40_000
MATCH_RATE = 0.12          # fraction of files with at least one candidate
COST_PER_FILE = 0.55       # USD to investigate one file at high reasoning effort
BATCH, CONCURRENCY = 5, 5  # --batch-size 5 --concurrency 5 => 25 files in flight


def funnel(files: int, match_rate: float, cost_per_file: float) -> tuple[int, float, float]:
    assert 0.0 <= match_rate <= 1.0 and files > 0
    investigated = round(files * match_rate)
    return investigated, investigated * cost_per_file, files * cost_per_file


inv, gated_cost, ungated_cost = funnel(FILES, MATCH_RATE, COST_PER_FILE)
print(f"[1] {FILES:,} files, {100 * MATCH_RATE:.0f}% carry a candidate -> {inv:,} investigated")
print(f"[1] cost with the regex funnel  = ${gated_cost:12,.2f}")
print(f"[1] cost investigating every file = ${ungated_cost:12,.2f}")
print(f"[1] the free scan stage avoids ${ungated_cost - gated_cost:,.2f} "
      f"({100 * (1 - MATCH_RATE):.0f}% of the spend)")
assert gated_cost < ungated_cost

# --- 2. Matcher recall is a hard ceiling on system recall ------------------
# A file with no candidate never reaches the agent, so the agent's own skill
# cannot recover a vulnerability the regex stage missed.
print()


def system_recall(matcher_recall: float, agent_recall: float) -> float:
    assert 0.0 <= matcher_recall <= 1.0 and 0.0 <= agent_recall <= 1.0
    return matcher_recall * agent_recall


for mr in (0.5, 0.7, 0.9):
    for ar in (0.8, 0.95):
        print(f"[2] matcher recall {mr:.2f} x agent recall {ar:.2f} "
              f"-> system recall {system_recall(mr, ar):.3f}")
# A perfect agent cannot exceed the matcher stage.
assert system_recall(0.7, 1.0) == 0.7
# Improving the agent has less leverage than improving matchers when the
# matcher stage is the weaker of the two.
gain_agent = system_recall(0.7, 0.95) - system_recall(0.7, 0.80)
gain_match = system_recall(0.85, 0.80) - system_recall(0.7, 0.80)
print(f"[2] +0.15 agent recall buys {gain_agent:.3f}; "
      f"+0.15 matcher recall buys {gain_match:.3f} "
      f"-> {'matchers' if gain_match > gain_agent else 'agent'} is the better investment here")
assert gain_match > gain_agent

# --- 3. Revalidation: pay again to halve the false positives --------------
# The project documents a 50%+ FP reduction at roughly the cost of `process`.
print()


def after_revalidation(tp: int, fp: int, fp_reduction: float) -> tuple[int, float, float]:
    assert 0.0 <= fp_reduction <= 1.0
    fp_left = fp * (1.0 - fp_reduction)
    return tp, fp_left, tp / (tp + fp_left)


TP, FP = 120, 480
print(f"[3] first pass:    {TP} TP, {FP} FP -> precision {TP / (TP + FP):.3f}")
for red in (0.50, 0.75):
    _, fp_left, prec = after_revalidation(TP, FP, red)
    print(f"[3] revalidate {100 * red:.0f}%: {TP} TP, {fp_left:.0f} FP -> precision {prec:.3f}")
_, _, p50 = after_revalidation(TP, FP, 0.50)
assert p50 > TP / (TP + FP), "revalidation must raise precision"
# Revalidation cannot raise recall: it only reclassifies findings that exist.
assert after_revalidation(TP, FP, 0.99)[0] == TP, "true positives are unchanged by revalidation"
# Targeting revalidation at high severity only is the documented advice; model it.
HIGH_FRACTION = 0.25
full = (TP + FP) * COST_PER_FILE
targeted = (TP + FP) * HIGH_FRACTION * COST_PER_FILE
print(f"[3] revalidating everything ${full:,.2f} vs high-severity only ${targeted:,.2f} "
      f"({100 * (1 - targeted / full):.0f}% cheaper, at the cost of FPs left in the tail)")
assert targeted < full

# --- 4. Resumability changes the cost of failure --------------------------
# Append-only records + atomic claiming mean an interrupted run re-does only
# the in-flight batch, not the whole scan.
print()
IN_FLIGHT = BATCH * CONCURRENCY


def wasted_on_crash(done: int, in_flight: int, resumable: bool) -> int:
    return in_flight if resumable else done + in_flight


for frac in (0.25, 0.90):
    done = round(inv * frac)
    r, nr = wasted_on_crash(done, IN_FLIGHT, True), wasted_on_crash(done, IN_FLIGHT, False)
    print(f"[4] crash at {100 * frac:.0f}% ({done:,} files done): "
          f"resumable wastes {r} files (${r * COST_PER_FILE:,.2f}), "
          f"non-resumable wastes {nr:,} (${nr * COST_PER_FILE:,.2f})")
assert wasted_on_crash(round(inv * 0.9), IN_FLIGHT, True) == IN_FLIGHT
# The saving grows with progress, which is exactly when a long scan is most
# likely to have hit a quota limit or a transient provider error.
assert (wasted_on_crash(round(inv * 0.9), IN_FLIGHT, False)
        > wasted_on_crash(round(inv * 0.25), IN_FLIGHT, False))

print("\nAll assertions passed.")

Executed output:

[1] 40,000 files, 12% carry a candidate -> 4,800 investigated
[1] cost with the regex funnel  = $    2,640.00
[1] cost investigating every file = $   22,000.00
[1] the free scan stage avoids $19,360.00 (88% of the spend)

[2] matcher recall 0.50 x agent recall 0.80 -> system recall 0.400
[2] matcher recall 0.50 x agent recall 0.95 -> system recall 0.475
[2] matcher recall 0.70 x agent recall 0.80 -> system recall 0.560
[2] matcher recall 0.70 x agent recall 0.95 -> system recall 0.665
[2] matcher recall 0.90 x agent recall 0.80 -> system recall 0.720
[2] matcher recall 0.90 x agent recall 0.95 -> system recall 0.855
[2] +0.15 agent recall buys 0.105; +0.15 matcher recall buys 0.120 -> matchers is the better investment here

[3] first pass:    120 TP, 480 FP -> precision 0.200
[3] revalidate 50%: 120 TP, 240 FP -> precision 0.333
[3] revalidate 75%: 120 TP, 120 FP -> precision 0.500
[3] revalidating everything $330.00 vs high-severity only $82.50 (75% cheaper, at the cost of FPs left in the tail)

[4] crash at 25% (1,200 files done): resumable wastes 25 files ($13.75), non-resumable wastes 1,225 ($673.75)
[4] crash at 90% (4,320 files done): resumable wastes 25 files ($13.75), non-resumable wastes 4,345 ($2,389.75)

All assertions passed.

The section 2 result is the one that should change behaviour. System recall is the product of matcher recall and agent recall, so when the matcher stage is the weaker link, adding matchers beats upgrading models. At the modelled point, buying 0.15 of matcher recall returns more than buying 0.15 of agent recall. This is why the project's documentation pushes users toward writing matchers rather than toward model selection, and it is why the dead regex hint found above is worth fixing even though its blast radius looks small.

How to run it in production

Treat the scanner as a coding agent with shell access. The project's own security model says exactly this: it is designed for trusted inputs, meaning your source code, but vendored code and external dependencies are a prompt-injection surface. Two isolation properties are worth understanding precisely, because they determine what a successful injection could achieve:

• Credentials are injected outside the sandbox, so a compromised worker cannot exfiltrate the model API keys.
• Worker egress is restricted to coding-agent hosts. Egress is open during bootstrap, but the agent does not run during bootstrap.

That combination bounds the damage but does not eliminate it: a worker still has your source tree. Run scans in the sandbox mode rather than on a developer workstation or a CI runner with broad credentials, and treat the finding output as untrusted text until reviewed.

Budget before you start, and stage the spend. A sensible order is scan (free) to size the candidate set, then a process run scoped to the highest-value paths, then wider passes. The configuration supports priorityPaths and ignorePaths for exactly this. Reserve revalidate for high-severity findings, which is the documented advice and which the cost model above quantifies.

Expect to stop on quota, and plan to resume. If a run halts because the upstream credential ran out of quota or credits, the tool stops and reports where to top up; re-running the same command continues. Because records are append-only and files are claimed atomically, this is safe with multiple workers in flight.

Wire the diff mode into CI separately from full scans. process --diff is the per-PR path. A full-repo scan is a periodic budgeted event, not a pipeline stage.

How to maintain it

• Re-run the matcher self-audit after every matcher change. It is free, it takes seconds, and it catches dead patterns like the one above. The project ships its own test for this; the harness here is a standalone equivalent that needs no workspace install.
• Track matcher recall against known findings. Every confirmed vulnerability that no matcher would have surfaced is a matcher to write.
• Re-pin the agent backend and model. Defaults move; gpt-5.5, claude-opus-4-8, and zai/glm-5.2 are the defaults at the audited commit, not stable contracts.
• Keep the repository-context document short. It is injected into every scan batch, so verbose context dilutes signal on every call and costs tokens on every call. The project's own bootstrap prompt targets 50 to 100 lines.
• Watch the false-positive rate as a trend, not a number. It is the metric that determines whether reviewers keep reading the output.

Failure modes

• Silent recall gap from a dead matcher. A pattern that can never fire produces no error and no finding. Demonstrated above with a real example.
• Applicability hidden inside match(). A broad filePatterns with a narrow internal gate makes coverage impossible to audit from the manifest.
• Budget exhaustion mid-scan. Mitigated by resumability, but only if you re-run rather than restart from a clean directory.
• Prompt injection from vendored code. The scanner reads whatever is in the tree. Sandbox mode is the mitigation, not an optional extra.
• Reviewer fatigue at 20% precision. A first pass without revalidation can be mostly false positives; shipping that to engineers burns the tool's credibility once.
• Mistaking the funnel for full coverage. A clean scan means no matcher fired and no agent objected. It does not mean the code is safe.
• Treating results as reproducible. Model-driven findings vary between runs; do not gate a build on exact-match output.
• Running it with production credentials in scope. It is an agent with shell access; give it a tree and nothing else.

Open questions and validation

• No scan was executed here. All cost, precision, and recall figures in the Python model are parameterized illustrations, and the real values depend entirely on the codebase and matcher set.
• The documented "50%+ FP reduction" from revalidation is the project's own empirical claim and was not independently reproduced.
• 118 of the 200 matcher modules could not be loaded without a workspace install, so the audit covers 82 modules and 681 examples, not the whole registry. A full audit requires pnpm install and running the project's own test suite.
• Whether matcher-first investment beats model-first investment in practice depends on where your true recall sits on each stage, which is measurable only against a corpus of known findings.

References

• deepsec repository, Apache-2.0: https://github.com/vercel-labs/deepsec
• deepsec project site: https://deepsec.sh
• Pipeline internals (docs/architecture.md): https://github.com/vercel-labs/deepsec/blob/main/docs/architecture.md
• Writing matchers (docs/writing-matchers.md): https://github.com/vercel-labs/deepsec/blob/main/docs/writing-matchers.md
• Model selection and defaults (docs/models.md): https://github.com/vercel-labs/deepsec/blob/main/docs/models.md
• Vercel Sandbox, used for distributed execution: https://vercel.com/docs/vercel-sandbox

Related: Agent security threat model · Offensive AI and the arms race · Anatomy of an autonomous agent intrusion · Agent sandboxing and isolation · Cybersecurity agent evaluation · Agent loop economics