Markdown

Risk-tiered human approval gates for agent actions¶

Scope: a graduated ladder of oversight, auto, validate, approve, multi-approve, that decides how much human involvement an agent action needs before it runs, and the concrete mechanisms that implement each rung. Distinct from agent intent verification (one specific mechanism, out-of-band signed-intent attestation, used to harden the approve and multi-approve rungs once an action is already known to need one) and from the agent policy engine (the binary permit/deny authorization decision a request must also pass, independent of how much human oversight it needs). The general form of the three-track pattern governing self-modifying agents already applies specifically to harness and prompt changes.

Vendor specs, paper results, and control-framework references below are quoted or paraphrased from the cited sources at research time; verify current status against the linked spec or paper before relying on it. The Python blocks are runnable and self-checking.

What it is¶

Most agent deployments default to one of two failure modes: every action pauses for a human, which agent intent verification already names as a failure mode in its own right, "users develop approval fatigue and rubber-stamp," or no action does, the "Excessive Agency" risk named across OWASP's LLM and agentic-application guidance: an agent taking high-impact actions, refunds, account changes, external data transfers, administrative actions, without appropriate human oversight.¹ A risk-tiered ladder is the alternative: classify each action by how much it could cost to get wrong, and route it to the cheapest oversight mechanism that actually bounds that cost.

Four rungs, cheapest to most expensive:

Auto. No gate. Logged, not blocked. For low-impact, reversible, non-sensitive actions, the default OWASP's least-agency principle implies: an agent should hold only the narrowest permissions its task needs, and within those permissions, routine reads and reversible writes should not spend a human's time.¹
Validate. An automated secondary check, a schema or invariant check, a policy re-evaluation, or a second model acting as a critic, runs synchronously with no human in the loop. It either passes (the action proceeds) or fails (the action escalates to Approve). There is no third outcome.
Approve. A single human must explicitly consent before the action runs, blocking and synchronous. MCP's elicitation mechanism and LangGraph's human-in-the-loop middleware are the two concrete, shipping implementations of this rung.
Multi-approve. At least two independent humans must consent, and the requester cannot be one of them: the maker-checker, four-eyes, segregation-of-duties control long used in financial and access-control settings, formalized in the COSO Internal Control Framework and in NIST SP 800-53's separation-of-duties control family, mapped onto agent actions whose blast radius or irreversibility justifies the extra friction.⁸

Why it matters¶

Tiering is not just theoretically tidy, it is measured to preserve most of the throughput a blanket policy would destroy. The GAIE framework, a three-tier graduated oversight model for agentic code generation in regulated domains, classifies tasks by regulatory impact, customer proximity, reversibility, and data sensitivity, and reports that graduated oversight "preserves 84-97% of agentic coding velocity (central estimate: 91%) while maintaining compliance evidence coverage for regulated functions."² That is the concrete cost of the alternative: gate everything and give up most of that velocity for coverage a risk-based ladder gets almost as well.

The classification itself needs a principled basis, not ad hoc per-feature thresholds. "Measuring AI Agent Autonomy" frames the problem along exactly two axes, impact (the potential effects of an agent's actions) and oversight (the degree of human supervision and control), and proposes assessing autonomy from an agent's orchestration code rather than only at runtime, so a system's oversight posture can be audited before it ever executes.³ AURA, a separate risk-assessment framework aimed at agentic AI deployments broadly, converges on the same shape from a different angle: a scored, quantified risk assessment with human-in-the-loop escalation built in, evidence that multiple independent research groups are arriving at the same structure rather than this being one lab's idiosyncratic preference.⁴ OpenAI's own governance guidance for agentic systems argues for calibrating the level of oversight to the actual risk of an action rather than requiring uniform human approval everywhere, with safe defaults that escalate to a human specifically when instructions are unclear or a situation was not anticipated.⁵

When to use it (and when not)¶

Use a full four-tier ladder when:

The agent's action space spans a wide range of consequence, from read-only lookups to irreversible, high-value, or regulated actions, so a single uniform policy (all-auto or all-approve) is wrong at one end no matter which end you pick.
Multiple people or teams can trigger the same action class, so an audit trail of who approved what, and proof no one approved their own request, actually matters.
The organization already has a defined risk taxonomy (regulatory exposure, customer proximity, reversibility, data sensitivity) the tiers can hang off, rather than inventing thresholds ad hoc per feature.

Skip or simplify it when:

The agent's entire action space is uniformly low-risk (read-only tools, a sandboxed dev environment with no production blast radius): auto for everything is correct, and four rungs is governance theater over a system that needed none.
The agent's entire action space is uniformly high-stakes (a single irreversible action is the whole product surface): collapse straight to approve or multi-approve and skip auto/validate, since there is no lower-risk population to route away from a human.
The team cannot staff a real second approver. Multi-approve with a rubber-stamp second reviewer, the same person under a different login, or someone who never actually reviews, is worse than approve alone: it manufactures the appearance of a control that is not there.

Architecture¶

flowchart TB
  ACT["Agent proposes an action"] --> CLASSIFY["Classify: reversibility, blast radius,<br/>data sensitivity, regulatory impact"]
  CLASSIFY -->|"low risk"| AUTO["Auto: logged, not blocked"]
  CLASSIFY -->|"moderate"| VALID["Validate: automated check"]
  VALID -->|"pass"| RUN["Action runs"]
  VALID -->|"fail"| ESC["Escalate"]
  CLASSIFY -->|"high"| APPROVE["Approve: one human<br/>(MCP elicitation / LangGraph interrupt)"]
  ESC --> APPROVE
  CLASSIFY -->|"critical"| MULTI["Multi-approve: 2+ distinct<br/>non-requester humans"]
  APPROVE -->|"consent"| RUN
  MULTI -->|"quorum reached"| RUN
  AUTO --> RUN

Classification happens once, before any rung; every rung below Auto either produces a RUN decision or an explicit escalation, never a silent stall and never a silent bypass.

How to classify an action into a tier¶

The classifier's job is to turn the risk factors above into one of four tiers, with a hard floor so soft factors can never talk an irreversible or regulated action down to Auto, the same non-negotiable-floor idea governing self-modifying agents already applies to its own hard-blocked track. This block scores an action on GAIE's blast-radius and data-sensitivity factors, then applies reversibility and regulatory impact as a floor rather than another additive term, and checks that the result is monotone (raising any risk factor never lowers the assigned tier):

# risk_tier_classifier.py -- validated: classify an agent action into one of four
# oversight tiers (auto / validate / approve / multi_approve) from risk factors
# modeled on GAIE's four classification factors (regulatory impact, customer
# proximity, reversibility, data sensitivity: arXiv 2606.22484) and the
# impact/oversight axes of "Measuring AI Agent Autonomy" (arXiv 2502.15212).
# A hard floor guarantees no combination of soft factors can downgrade an
# irreversible or regulatory-impact action down to auto. Standard library only.

TIERS = ["auto", "validate", "approve", "multi_approve"]

def classify_tier(reversible: bool, blast_radius: int, data_sensitivity: int,
                   regulatory_impact: bool) -> str:
    """blast_radius and data_sensitivity are each 0..3 (none/low/medium/high).
    A combined score in 0..6 picks a baseline tier; a hard floor then raises
    (never lowers) the tier for irreversible or regulatory-impact actions."""
    assert 0 <= blast_radius <= 3 and 0 <= data_sensitivity <= 3
    score = blast_radius + data_sensitivity
    if score <= 1:
        baseline = "auto"
    elif score <= 3:
        baseline = "validate"
    elif score <= 5:
        baseline = "approve"
    else:
        baseline = "multi_approve"

    floor = "approve" if (not reversible or regulatory_impact) else "auto"
    return TIERS[max(TIERS.index(baseline), TIERS.index(floor))]

# Happy path: a harmless, reversible, non-sensitive, non-regulatory read runs auto.
assert classify_tier(reversible=True, blast_radius=0, data_sensitivity=0, regulatory_impact=False) == "auto"

# A moderate-impact, reversible action (e.g. drafting an email, not sending it)
# lands on validate: an automated check, no human blocking the happy path.
assert classify_tier(reversible=True, blast_radius=1, data_sensitivity=1, regulatory_impact=False) == "validate"

# A high-impact but still reversible action (e.g. reconfiguring a non-prod service)
# requires a single human approval.
assert classify_tier(reversible=True, blast_radius=2, data_sensitivity=3, regulatory_impact=False) == "approve"

# A high-impact, sensitive, irreversible action (e.g. a large wire transfer)
# requires multi-party approval.
assert classify_tier(reversible=False, blast_radius=3, data_sensitivity=3, regulatory_impact=True) == "multi_approve"

# Adversarial: the hard floor. An action with a LOW combined score (would baseline
# to auto) but that is irreversible must NEVER be downgraded to auto or validate.
low_score_irreversible = classify_tier(reversible=False, blast_radius=0, data_sensitivity=0,
                                        regulatory_impact=False)
assert low_score_irreversible == "approve", f"irreversibility must floor at approve, got {low_score_irreversible}"

# Adversarial: regulatory impact alone floors the same way, even with a trivial score.
low_score_regulatory = classify_tier(reversible=True, blast_radius=0, data_sensitivity=0,
                                      regulatory_impact=True)
assert low_score_regulatory == "approve", f"regulatory impact must floor at approve, got {low_score_regulatory}"

# Monotonicity: for fixed reversible/regulatory flags, raising blast_radius or
# data_sensitivity must never LOWER the assigned tier.
for reversible in (True, False):
    for regulatory in (True, False):
        prev = None
        for score_pair in [(0, 0), (1, 0), (1, 1), (2, 1), (2, 2), (3, 2), (3, 3)]:
            tier = classify_tier(reversible, score_pair[0], score_pair[1], regulatory)
            idx = TIERS.index(tier)
            if prev is not None:
                assert idx >= prev, "raising risk factors must never lower the tier"
            prev = idx

print("OK: baseline tiers auto/validate/approve/multi_approve all reachable; "
      "irreversibility and regulatory impact each floor at approve regardless of "
      "score; tier assignment is monotone in blast_radius and data_sensitivity")

Run output:

OK: baseline tiers auto/validate/approve/multi_approve all reachable; irreversibility and regulatory impact each floor at approve regardless of score; tier assignment is monotone in blast_radius and data_sensitivity

How to implement Approve: MCP elicitation and LangGraph interrupts¶

The Model Context Protocol's elicitation mechanism is the concrete, standardized way an MCP server pauses a tool call and asks the user to consent before it proceeds. A server sends an elicitation/create request in form mode with a restricted JSON Schema (flat objects, primitive properties only), and the client's response is always exactly one of three actions: accept (with the submitted data), decline (explicit no), or cancel (dismissed without a choice), so a caller can never mistake "the user closed the dialog" for "the user said no."⁶ The spec is explicit that this channel must never carry secrets: "Servers MUST NOT use form mode elicitation to request sensitive information such as passwords, API keys, access tokens, or payment credentials," reserving a separate URL mode, an out-of-band browser redirect, for anything that sensitive.⁶

// Server asks for explicit consent before a destructive action, e.g. deleting
// a resource; the client MUST render this as a reviewable form, not auto-accept it.
{
  "method": "elicitation/create",
  "params": {
    "mode": "form",
    "message": "Confirm: delete 'quarterly-report.pdf'? This cannot be undone.",
    "requestedSchema": {
      "type": "object",
      "properties": { "confirm": { "type": "boolean", "title": "Delete file" } },
      "required": ["confirm"]
    }
  }
}

// Client's response is exactly one of three actions; "cancel" is not "decline".
{ "action": "accept", "content": { "confirm": true } }

LangGraph's human-in-the-loop middleware is the equivalent for a graph-based harness rather than an MCP tool boundary: it checks each proposed tool call against a configurable policy, and when a call needs review it issues an interrupt that pauses the graph, persists state through a checkpointer, and resumes on one of four decisions, approve, edit (modify the call before it runs), reject (with feedback), or respond (answer an "ask user" tool directly).⁷ Both mechanisms are the plumbing for the Approve rung; when the stakes are high enough that the consent itself must be unforgeable against a compromised chat channel, harden it with out-of-band signed-intent attestation rather than trusting an in-band accept.

How to implement Validate and Multi-approve¶

The property that matters for Validate is that it never has a third outcome: a passing check proceeds with no human touched, a failing check escalates to Approve, and there is no path where a failed check silently proceeds or silently blocks forever with nothing to resume it. The property that matters for Multi-approve is the maker-checker invariant itself: the requester can never be counted among the required approvers, and a single approver "approving" more than once must not be double-counted as two distinct signers, exactly the collusion-of-one loophole segregation-of-duties controls exist to close.⁸ This block validates both:

# approval_gates.py -- validated: the Validate tier auto-escalates to Approve on
# a failed automated check rather than silently blocking forever or silently
# proceeding; the multi_approve tier enforces a maker-checker invariant (the
# requester can never be counted as one of the required distinct approvers),
# the segregation-of-duties control COSO and NIST 800-53 require in financial
# and access-control settings, mapped onto agent actions. Standard library only.

def run_validate_gate(action, automated_check) -> str:
    """Returns 'proceed' if the automated check passes, otherwise escalates to
    the approve tier. There is no third outcome: a check always resolves to
    exactly one of these two paths."""
    return "proceed" if automated_check(action) else "escalate_to_approve"

class MultiApproveGate:
    """Maker-checker: the requester cannot also be an approver, and the action
    is authorized only once enough DISTINCT non-requester approvers sign off."""
    def __init__(self, required_approvers: int, requester: str):
        assert required_approvers >= 2, "multi_approve requires at least two approvers"
        self.required = required_approvers
        self.requester = requester
        self.approvals = set()

    def approve(self, approver: str) -> None:
        if approver == self.requester:
            raise PermissionError("requester cannot approve their own action (maker-checker)")
        self.approvals.add(approver)

    def is_authorized(self) -> bool:
        return len(self.approvals) >= self.required

# Validate gate: a passing automated check proceeds with no human involved.
assert run_validate_gate("draft-email", automated_check=lambda a: True) == "proceed"
# A failing automated check escalates rather than silently blocking or proceeding.
assert run_validate_gate("draft-email", automated_check=lambda a: False) == "escalate_to_approve"

# Multi-approve happy path: two distinct, non-requester approvers authorize the action.
gate = MultiApproveGate(required_approvers=2, requester="agent-alice")
assert not gate.is_authorized()          # zero approvals so far
gate.approve("finance-lead-bob")
assert not gate.is_authorized()          # one approval is not enough
gate.approve("controller-carol")
assert gate.is_authorized()              # two distinct approvers: authorized

# Adversarial: the requester cannot approve their own action.
gate2 = MultiApproveGate(required_approvers=2, requester="agent-alice")
try:
    gate2.approve("agent-alice")
    raised = False
except PermissionError:
    raised = True
assert raised, "self-approval must be rejected"

# Adversarial: one person "approving" twice (a double click, or a compromised
# session replaying its own approval) must not count as two distinct approvers.
gate3 = MultiApproveGate(required_approvers=2, requester="agent-alice")
gate3.approve("controller-carol")
gate3.approve("controller-carol")        # same person again
assert len(gate3.approvals) == 1
assert not gate3.is_authorized(), "a single approver replaying their own approval must not authorize"

# Boundary: exactly meeting the required count authorizes; one short does not.
gate4 = MultiApproveGate(required_approvers=3, requester="agent-alice")
for approver in ("bob", "carol"):
    gate4.approve(approver)
assert not gate4.is_authorized()         # 2 of 3
gate4.approve("dave")
assert gate4.is_authorized()             # 3 of 3

print("OK: validate gate always resolves to proceed/escalate, never a third state; "
      "multi_approve needs 2+ distinct non-requester approvers; self-approval "
      "raises; a replayed single-approver signature does not authorize")

Run output:

OK: validate gate always resolves to proceed/escalate, never a third state; multi_approve needs 2+ distinct non-requester approvers; self-approval raises; a replayed single-approver signature does not authorize

Failure modes¶

Over-gating into approval fatigue. Every action, including harmless reads, demanding a human click; agent intent verification already names the outcome, users rubber-stamp, and the gate stops meaning anything.
Under-gating into excessive agency. No tier above Auto exists at all, the OWASP failure mode this page's ladder is built to prevent, particularly for refunds, account changes, external data transfers, and administrative actions.¹
A Validate step with a third, silent outcome. A failed automated check that proceeds anyway, or that blocks with no escalation path and no one ever notices the stall; the code above tests that only proceed and escalate_to_approve exist.
A Multi-approve gate satisfied by one person. The requester approving their own action, or a single approver's signature counted twice, both defeat the entire point of a second, independent reviewer; both are adversarially tested above.
A tier classification that never gets re-validated. Set once at launch and never revisited as new tools or action types are added; a new, higher-risk action silently inherits the most permissive default rather than being explicitly classified.
MCP form-mode elicitation used to collect a secret. Explicitly forbidden by the spec; route anything that sensitive through URL mode instead.⁶

Open questions & validation¶

Whether your risk classification's hard floor actually holds under adversarial testing: try to construct an irreversible, regulated action that still scores into Auto, the way the code above tests it, and confirm it cannot.
Whether your Multi-approve gate is enforced server-side as a real invariant in code, or is a process convention two people are merely trusted to follow by hand.
Whether the Validate tier's automated check is itself adversarially tested; a check that always returns true is not a gate, it is a rubber stamp with extra steps.
Whether tier assignment is reviewed when a new tool or action type is added, or silently inherits Auto by omission.
Whether MCP or LangGraph's shipped mechanisms are actually wired end to end from your risk classifier to the Approve rung, not merely available as a library feature nobody calls.

References¶

Model Context Protocol, Elicitation specification (form mode, URL mode, the accept/decline/cancel response model): https://modelcontextprotocol.io/specification/draft/client/elicitation
LangChain, Human-in-the-Loop middleware for LangGraph (approve/edit/reject/respond, checkpointer-backed interrupts): https://docs.langchain.com/oss/python/langchain/human-in-the-loop
OWASP, Top 10 for Agentic Applications for 2026: https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/
OWASP, Top 10 for LLM Applications (Excessive Agency, LLM06:2025): https://genai.owasp.org/llm-top-10/
Kang, "Governed AI-Assisted Engineering: Graduated Human Oversight for Agentic Code Generation in Regulated Domains" (GAIE): https://arxiv.org/abs/2606.22484
Chiris and Mishra, "AURA: An Agent Autonomy Risk Assessment Framework": https://arxiv.org/abs/2510.15739
Cihon, Stein, Bansal, Manning, and Xu, "Measuring AI Agent Autonomy": https://arxiv.org/abs/2502.15212
Shavit, Agarwal, et al., "Practices for Governing Agentic AI Systems" (OpenAI): https://openai.com/index/practices-for-governing-agentic-ai-systems/
COSO, Internal Control - Integrated Framework (segregation of duties as a control activity): https://www.coso.org/guidance-on-ic
NIST SP 800-53 Rev. 5 (separation/segregation of duties control family): https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final

OWASP's Top 10 for LLM Applications names "LLM06:2025 Excessive Agency," an agent taking high-impact actions without appropriate human oversight, and recommends human-in-the-loop approval for high-impact actions plus minimum-necessary permissions. https://genai.owasp.org/llm-top-10/ The separate, 2026 Top 10 for Agentic Applications extends this into agent-specific "ASI" risk categories and (per secondary sources, not independently confirmed against the primary document, which is access-gated) a named "least agency" principle; the specific list of sensitive operations this page names (refunds, account changes, external data transfers, administrative actions) is this page's own illustrative framing of the excessive-agency risk, not a verbatim quotation from either OWASP document. https://genai.owasp.org/resource/owasp-top-10-for-agentic-applications-for-2026/ ↩↩↩
Kang, "Governed AI-Assisted Engineering" (GAIE): a three-tier graduated human oversight model, human-in-the-loop for strategic functions, human-over-the-loop for customer-impacting work, automated-with-monitoring for internal work, classifying tasks by regulatory impact, customer proximity, reversibility, and data sensitivity; reports 84 to 97 percent (91 percent central estimate) agentic coding velocity preserved while maintaining compliance evidence coverage for regulated functions. https://arxiv.org/abs/2606.22484 ↩
Cihon, Stein, Bansal, Manning, and Xu, "Measuring AI Agent Autonomy": proposes assessing an AI agent's autonomy along two axes, impact (the potential effects of its actions) and oversight (the degree of human supervision and control), via a code-based method that evaluates an agent's orchestration code (demonstrated on the AutoGen framework) rather than relying only on runtime evaluation. https://arxiv.org/abs/2502.15212 ↩
Chiris and Mishra, "AURA: An Agent Autonomy Risk Assessment Framework": a unified framework to detect, quantify, and mitigate risks from agentic AI systems, emphasizing human-in-the-loop oversight through agent-to-human communication mechanisms across both synchronous and asynchronous operation. https://arxiv.org/abs/2510.15739 ↩
Shavit, Agarwal, and co-authors, "Practices for Governing Agentic AI Systems" (OpenAI): argues for calibrating the level of human oversight to an action's actual risk rather than uniform approval requirements, with safe defaults that escalate to human review when instructions are unclear, alongside assigning agent identities, logging decision trails, and defining documented human escalation points. https://openai.com/index/practices-for-governing-agentic-ai-systems/ ↩
Model Context Protocol, Elicitation: a server pauses a tool call with an elicitation/create request; form mode collects structured data via a restricted JSON Schema (flat objects, primitive properties only) and MUST NOT be used for passwords, API keys, access tokens, or payment credentials (URL mode, an out-of-band browser redirect, exists for that); every response is exactly one of three actions, accept (with submitted data), decline (explicit refusal), or cancel (dismissed without a choice). https://modelcontextprotocol.io/specification/draft/client/elicitation ↩↩↩
LangChain, Human-in-the-Loop middleware for LangGraph: checks each proposed tool call against a configurable policy; a call needing review issues an interrupt that pauses the graph, with state persisted via a checkpointer (e.g. AsyncPostgresSaver in production, InMemorySaver for testing) so execution can resume later on one of four human decisions, approve, edit, reject, or respond. https://docs.langchain.com/oss/python/langchain/human-in-the-loop ↩
The four-eyes / maker-checker / segregation-of-duties control requires that no single individual both initiates and approves a consequential action; formalized as a control activity in the COSO Internal Control - Integrated Framework and as a separation-of-duties control family in NIST SP 800-53, used across financial and access-control settings specifically so that abuse requires collusion between independent parties rather than one person's unilateral decision. https://www.coso.org/guidance-on-ic ; https://csrc.nist.gov/pubs/sp/800/53/r5/upd1/final ↩↩