Two people at a desk reviewing and annotating a document by hand beside laptops

Human in the Loop AI Governance: Designing Effective Approval Workflows

July 22, 2026 7 min read

Adding a human approval step to an agent workflow does not make it safe. It makes it as safe as the approver's attention, and attention is the first thing a high-volume approval queue wears down. A governance design has to decide which actions need a person, what that person sees, what exactly their approval authorizes, and what is recorded afterwards.

This guide takes a position that differs from most advice on the topic. Gate by what an action can do, not by how confident the model says it is. Show the reviewer the concrete change before the model's explanation. Make an approval a single-use permission bound to one action, not a general go-ahead.

Where agents need a gate

Connect a language model to a production database or an external API and it will eventually issue a malformed request, invent a parameter, or act outside its intended scope. Three failure classes account for most of the cases that matter.

Drift. Over a long multi-step run, early constraints get diluted by intermediate results, and an agent auditing records begins editing them because a retrieved document hinted at a correction.
Injected instructions. Untrusted input such as a customer email or an uploaded file can carry instructions that override the system prompt. Without an independent approval layer the agent may execute them.
Plausible wrong parameters. A tool call can be perfectly formed and still wrong: a valid instance type in the wrong region, a refund to the wrong account.

An approval gate sits between the model's proposal and the side effect. It fixes none of these failures. It gives a person a chance to catch the consequence.

Gate by action risk, not by model confidence

A popular pattern has the model report a confidence score and routes low-confidence actions to a human. The scores are not reliable enough to carry that decision. The study Can LLMs Express Their Uncertainty? found that language models tend to be overconfident when they state their confidence in words, and that calibration depends on model capability. A model that is wrong in an action it should not take is likely to be confident about it.

Classify the tool instead, and decide what an unclassified tool is. The Model Context Protocol schema shows the trap. Its tool annotations (readOnlyHint, destructiveHint and others) default to readOnlyHint: false and destructiveHint: true when a server omits them, which is the cautious direction. But the same schema says all annotation properties are hints and that clients should never make tool-use decisions based on them from untrusted servers, so the risk tier cannot come from the server's own label. Keep the tier in your own registry, set by an owner or derived from the tool type and HTTP method. Treat any write you cannot classify as high impact. Run one deterministic policy check immediately before the side effect, covering explicit deny, read-only ceilings, required scopes, argument limits such as a maximum amount, and an identity requirement. Irreversible, financial, external-communication and privileged actions always require confirmation. Because the rule is a function of the tool and its arguments, it can be tested and audited, and it does not drift with the model.

Dynamic triggers fit the same mold: require approval when the amount exceeds a threshold, when the recipient is external, or when the target record belongs to a customer rather than the requester. The trigger inspects parameters in code before deciding to pause.

Show the change, then the reasoning

Common advice is to give the reviewer the agent's full reasoning trace so they can judge the decision. Evidence suggests the explanation can work against you. In a study of human-AI decision-making, Does the Whole Exceed its Parts? found that explanations increased the chance people accept the AI's recommendation, regardless of whether it was correct. A fluent rationale from the model is an argument for approving.

Design the review screen so the verifiable parts come first:

The exact action: the tool, the target record, and the parameters, as a before-and-after diff where one exists.
The evidence the agent retrieved, with links to the sources, so the reviewer can check the input.
The requester and the original request, so the reviewer sees whether the action matches what was asked.
The agent's explanation last, collapsed by default.

Also write reviewer-facing text for the parameters that matter. "Approve API call: POST /v1/billing" tells nobody anything. "Refund 750 GBP to the card ending 4242 on order A184" does.

Bind each approval to one action

An approval that can be reused, replayed, or applied to a different action is a standing permission. Treat it as a ticket for one thing. The pending action should be an immutable snapshot of what will run, and the confirmation should bind to a hash of those arguments and to the app, conversation, and verified requester that proposed it. State changes should use compare-and-set so that at most one confirmation wins. Give it a short expiry. Our companion guide on binding human approval to exact AI agent actions covers the protocol in detail.

Approval screens also fail through habit. Anthropic's engineering post on Claude Code auto mode reports that "over time that leads to approval fatigue, where people stop paying close attention to what they're approving", and that "in practice users accept 93% of them anyway". A confirmation that people accept 93% of the time protects very little unless it is rare and specific. Ask for approval only on writes that clear a risk tier, show the exact item and the exact change, and accept only an unambiguous yes. A reply with a condition, a second request or a question, such as "yes but add a thanks line", is not an approval of the original text. Expire the pending record, and make the move from waiting to sending a compare-and-set so an approval releases the action once. A long-running automation that must wait for a person should therefore be a workflow with an explicit human step, not a single run holding a pending prompt that expires.

Implement the pause

A reviewer works on a different timescale from the agent, so the workflow cannot hold a thread open. Persist the state (context, proposed action, evidence), suspend, and resume on the decision event.

def execute_tool_with_governance(tool, args, ctx):
    decision = policy.evaluate(tool, args, ctx)        # deterministic, no model
    if decision == "DENY":
        return deny(decision.reason)
    if decision == "REQUIRE_CONFIRMATION":
        pending = pending_actions.create(
            snapshot=canonical(tool, args),
            args_hash=hash_args(args),
            scope=(ctx.app, ctx.conversation, ctx.subject),
            expires_in=timedelta(minutes=30),
        )
        notify_reviewer(pending, role=required_role(tool))
        return "PAUSED_AWAITING_APPROVAL"
    return run(tool, args)

def on_decision(pending_id, reviewer, verdict, edited_args=None):
    pending = pending_actions.claim(pending_id)        # compare-and-set
    require(reviewer.can_approve(pending.tool))
    require(not policy.needs_second_person(pending.tool) or reviewer != pending.requester)
    if verdict == "approve" and hash_args(pending.args) == pending.args_hash:
        run(pending.tool, pending.args)
    audit.record(pending, reviewer, verdict)
    resume_workflow(pending, verdict, reviewer_feedback=edited_args)

Route each request by role: a schema change goes to a database administrator, a mass email to a marketing owner. Where the policy demands it, the reviewer must differ from the requester. On rejection, hand the reason back to the agent so it can try a safer route. Offer Modify as well as Approve and Reject, because a corrected parameter is better evidence than a bare refusal.

Plan for approval fatigue

The fastest way to defeat a gate is to send it too many requests. Reviewers who see hundreds of near-identical prompts start approving by reflex, and the OWASP agentic threat guide lists "Overwhelming Human in the Loop" as a threat in its own right, since an attacker can flood reviewers with low-risk requests to hide a harmful one. Put the controls where the volume is.

Filter first with deterministic checks: schema validation, argument limits, allow lists. A person should never see a request that code can reject.
Auto-run low-risk reversible actions and keep the queue for the tiers that matter.
Batch near-identical requests into one review with the list of targets and the shared rule.
Measure the approve rate and time-to-decision per reviewer and per tool. A reviewer who approves 100 percent in two seconds is not reviewing.
Sample some approvals for a second look, and occasionally plant a known-bad request in a test queue to see whether it is caught.

Latency is the other cost. A synchronous chat product often cannot wait on a person. There, limit the agent to read-only tools or to reversible actions the user can undo, and reserve approval for asynchronous work such as provisioning, bulk edits, and outbound messages.

Test the gate as an adversary

Write tests that try to reach a gated outcome through an ungated tool. If delete_user needs approval, check whether update_user can scramble the same record, whether a batch endpoint wraps the same delete, or whether a prompt injected through a ticket can talk the agent into the second path. Check replay too: an approval token must fail on a second use, after expiry, and for modified arguments.

Log each decision with the requester, tool, parameters, policy version and reason code, reviewer identity, timestamp, verdict, and what actually executed. That record satisfies an auditor, and it is also a dataset of the cases where the model was wrong and a person corrected it. Exclude secrets and raw sensitive content from it.

Next actions

List every tool your agent can call and assign each a risk tier. Make unlabeled writes default to the strictest review tier. Move one irreversible action behind a single-use approval bound to its arguments, build the review screen with the diff first and the explanation last, and track approve rate and decision time from day one.

References

Can LLMs Express Their Uncertainty? reports that models tend to be overconfident when verbalizing confidence.
Does the Whole Exceed its Parts? The Effect of AI Explanations on Complementary Team Performance reports that explanations raised acceptance of AI recommendations regardless of correctness.
OWASP Agentic AI threats and mitigations lists overwhelming the human in the loop as a threat.
Model Context Protocol schema, ToolAnnotations for the annotation defaults and the warning that they are hints from possibly untrusted servers.
Anthropic, Claude Code auto mode for approval fatigue and the 93% acceptance figure.

About Fire In Belly: Independent senior engineering from Tallinn, Estonia. We design and build AI workflow automation with the approval gates, audit trails, and human oversight described above, at published fixed prices. Schedule a call to discuss your next project.