Designing human-in-the-loop AI systems

Human-in-the-loop does not mean adding an “approve” button after an AI output. It means assigning people a clear role, enough evidence and time to exercise judgment, authority to change the outcome, and a route for escalation. A ceremonial reviewer who cannot detect an error or is expected to approve hundreds of items per minute is not a meaningful control.

The right review point depends on consequence, uncertainty, reversibility, and scale. Use people where judgment changes risk, and use deterministic checks where software can enforce a rule more consistently.

The NIST AI Risk Management Framework appendix on human-AI interaction highlights the importance of clearly defining and differentiating human roles and responsibilities. The CLEAR framework translates that principle into Consequence, Legibility, Empowerment, Attention, and Record.

Consequence: decide which outcomes deserve review

Map what happens if the output is wrong. Consider impact on the user, affected people, the organization, and downstream systems. Then consider whether the action can be reversed.

A suggested internal draft is usually easier to correct than a sent message. A proposed database change is easier to inspect than an executed deletion. A ranked list shown to an analyst differs from an automated denial of service. The model may be identical; the action boundary changes the risk.

Create risk tiers for your context:

Tier 0, assistive: private brainstorming, formatting, or suggestions with no automatic external action.
Tier 1, reversible: low-impact drafts or classifications with easy correction and monitoring.
Tier 2, consequential: communications, financial operations, access changes, or decisions that can materially affect someone.
Tier 3, critical or regulated: uses where qualified judgment, legal duties, safety, or fundamental rights require specialized governance.

These are not legal categories. Adapt them with domain experts and applicable obligations. Higher tiers should generally require stronger evidence, narrower automation, clearer authority, and more independent oversight.

Legibility: show the reviewer what matters

A reviewer needs more than polished prose. Design a review packet containing:

the original request or a faithful, access-controlled view;
the proposed output or action;
source passages with identifiers and dates;
tools called and important results;
validation failures, conflicts, or missing information;
the policy or rubric the reviewer applies;
what approval will cause.

Do not display a made-up confidence percentage as if it were calibrated certainty. If you have a confidence measure, document how it was produced and validate whether it predicts error on the current task. Concrete signals such as a missing source, conflicting values, a changed version, or a failed schema are often more actionable.

Reduce automation bias by making disagreement easy. Avoid interfaces that visually celebrate the generated answer while hiding source text behind several clicks. For comparisons, align each material claim with its evidence. For extracted fields, highlight the source region. For code, show a diff and tests rather than only an explanation.

Empowerment: give humans real authority

Define what each role can do:

Reviewer: approve, reject, edit, or request more evidence.
Escalation owner: resolve ambiguous or high-impact cases.
System owner: change prompts, tools, thresholds, or routing.
Risk or domain owner: define prohibited uses and acceptance criteria.
Incident owner: stop automation, preserve evidence, and coordinate response.

Approval should not be the only path. A reviewer needs a safe “cannot determine” state. The system should never punish escalation through impossible productivity targets. If a decision requires a licensed or specifically accountable professional, a generic crowd reviewer is not an equivalent substitute.

Clarify whether edits are merely feedback or immediately become an external action. Show the final payload before execution. Use two-person approval for selected irreversible or high-value operations where your risk analysis supports it.

Attention: design for realistic workload

Review quality falls when queues grow, cases are repetitive, or interfaces demand constant vigilance. Sample easy cases for quality measurement, but route ambiguous and high-impact cases deliberately. Rotate tasks, set workload limits, and monitor time-to-review alongside error detection.

Do not use the model’s own confidence as the only routing signal. Combine deterministic conditions and observed risk signals:

require_review when:
  action_value > approved_limit
  OR source_conflict = true
  OR required_evidence_missing = true
  OR output_validation_failed = true
  OR user_requests_human = true
  OR random_quality_sample = true

Random sampling matters because a routing rule cannot discover failures it was never designed to flag. Review both accepted and rejected outputs. Measure whether reviewers agree and investigate cases where the rubric itself is unclear.

Record: preserve an auditable decision trail

For each consequential review, retain the minimum appropriate record:

model and system configuration identifiers;
immutable input and output hashes or protected snapshots;
sources and versions presented;
automatic checks and routing reason;
reviewer identity and role;
approve, edit, reject, or escalate outcome;
final action and timestamp;
later correction or appeal.

Protect these records according to their sensitivity and retention requirements. An audit trail should not become a permanent warehouse of unnecessary personal data. Use access controls and redaction, and document retention and deletion.

The broader NIST AI RMF treats governance, contextual mapping, measurement, and management as connected lifecycle work. Review logs should feed that loop: observed failures change the evaluation set, rubric, controls, or automation boundary.

A worked example: refund assistance

Suppose an assistant reads a support request, retrieves order data and policy passages, then proposes a refund. The system must not let text in the customer message grant itself authority.

Tier 1 cases might be a draft reply with no account action. Tier 2 cases might propose a refund below a defined amount when the order record and policy agree. Review is required when the amount exceeds the limit, identity verification is missing, policy passages conflict, the customer alleges fraud, or a tool returns incomplete data.

The reviewer sees the request, verified order facts, applicable policy passages, proposed amount, and the exact action that approval will trigger. They can edit, reject, request evidence, or escalate to a fraud specialist. The audit record stores the evidence identifiers and final decision. A later dispute can be reconstructed without claiming the model “decided” alone.

Measure the complete review loop

Useful measures include:

proportion routed to review and why;
reviewer agreement on the same cases;
edits, rejections, and escalations by failure type;
errors discovered after approval;
time pressure and queue age;
appeal or correction rate;
difference between reviewed and unreviewed populations;
incidents prevented by deterministic checks versus human judgment.

A falling review rate is not automatically improvement. It might indicate better automation, a broken trigger, or users abandoning the workflow. Pair operational metrics with sampled quality checks.

CLEAR launch checklist

Consequence: Outcomes are risk-tiered, affected parties are mapped, and irreversible actions have explicit boundaries.

Legibility: Reviewers receive the original context, evidence, uncertainty signals, rubric, and effect of approval.

Empowerment: Named roles can edit, reject, stop, and escalate; authority matches the domain.

Attention: Queue volume is sustainable, routing uses several signals, and random sampling detects unknown failures.

Record: Decisions are reproducible, protected, retained proportionately, and connected to corrections.

Human review is valuable when it is a designed part of the control system. Place it before consequential actions, make evidence visible, and give the reviewer both authority and time. Otherwise the person becomes a signature added to automation rather than a meaningful safeguard.

Designing Human-in-the-Loop AI Systems