Human-in-the-Loop Design for AI Agents

"Human in the loop" is often written on slides as a reassurance. In practice it is a design discipline. A badly designed review step either slows everything down or turns into a button people click without reading, and then it protects nobody.

Decide what actually needs a human

Start by classifying the actions an agent might take:

• Reversible and low-impact, such as drafting text, tagging a ticket or suggesting a category. These can often run automatically with sampling for quality.
• Reversible but visible, such as sending an internal notification. Consider an approval step at first, relaxing it once the evaluation record supports that.
• Irreversible or high-impact, such as payments, external messages, record deletion or legal commitments. Require explicit approval, and enforce it in code so the agent cannot skip it.

Write this classification down and review it periodically. It is the backbone of your oversight model.

Choose the right pattern

Several patterns recur:

1. Approve before acting. The agent proposes; a person confirms. Suited to high-impact actions.
2. Review after acting, with sampling. The agent acts; a sample of runs is reviewed later. Suited to low-impact, high-volume tasks.
3. Escalate on uncertainty. The agent acts alone when conditions are clear and hands over when a rule fires, confidence is low or information is missing.
4. Collaborate. The person and the agent work through a task together, the agent preparing and the person deciding. Suited to analysis and drafting.

Most real systems combine these.

Make review fast and informed

The reviewer should see on one screen:

• The proposed action in plain language.
• The evidence behind it: source documents, retrieved passages, the data used.
• What the agent was unsure about, if anything.
• Simple controls to approve, edit or reject, and to say why.

If the reviewer has to open five systems to verify a proposal, review will be slow or perfunctory. Bringing evidence to the reviewer is one of the highest-value pieces of engineering in an agent project.

Guard against automation bias

People tend to trust automated suggestions, particularly when they are usually right. Counter that tendency deliberately:

• Show reasons and sources rather than just a recommendation.
• Occasionally include cases where the right answer is to reject, known to the oversight team but not the reviewer, and measure how they are handled.
• Track approval rates and review times. Near-instant approvals of every case are a warning sign.
• Rotate reviewers and avoid making review a rushed afterthought at the end of a shift.

Close the feedback loop

Every correction is information. Capture what was changed and why, add it to the evaluation set and review patterns regularly. If the same correction keeps appearing, fix the prompt, the tool or the retrieval, rather than relying on people to keep repairing the output.

Define accountability

Decide who is accountable when an approved action turns out to be wrong. Reviewers need clear authority, clear guidance on what to check and the ability to escalate. The organisation needs an incident process: how to pause the agent, how to investigate using the logs and how to correct affected records.

Respect users and affected people

When an agent's output affects customers, citizens or staff, consider how they will know automation was involved, how they can contest a decision and which decisions should always have a person behind them. In some jurisdictions and sectors there are legal requirements here, so involve your compliance team early.

Start conservative, relax with evidence

A sensible rollout begins with approval on every action, measures how often reviewers change the agent's proposal, and relaxes the requirement for categories where the record is consistently good. Never relax it for the categories you classified as irreversible. The aim is not to eliminate people from the workflow but to put their attention where it matters most.