Human-in-the-loop AI automation: where the person belongs.
Human-in-the-loop AI automation is a design in which the system does the repetitive work on its own and a person steps in only at pre-decided points. There are four: below a confidence threshold, before any irreversible action, on a sample of what the machine did, and at a kill switch. Done well, the person handles exceptions with the machine's notes attached; done badly, the person approves everything and nothing was automated.
This guide sets out the four places to put a person, how the six production systems Paul Rahme (Aldenebai) runs at a US marina-software company do it, the common mistakes, and a checklist to take into a vendor meeting.
The four places to put a person in an automated system
Everything else is either fully automated or not automated at all. The skill is choosing which of the four applies to each step.
Below the confidence threshold
The system scores its own output against the evidence it used. Above the line it acts; below it, it stops and escalates to a person with the intent, the sources, the draft, and the reason it stopped. The line is a dial the team owns; raising it trades autonomy for safety. Definition →
Before any irreversible action
Merging code, issuing a refund, changing an account, deleting data, moving money. These are not on the machine's allowed-action list at all; the machine prepares the case and a person presses the button. The gate is enforced by permissions, not by a prompt.
On a sample of what the machine did
A person reads a sample of the auto-closed tickets, the healed tests, the enriched records, and every escalation, especially in the first weeks. The audit trail makes this possible: every decision is logged with its reason, so a wrong result can be traced to a wrong source or a wrong threshold and fixed at the cause.
At the kill switch
One control that stops the system instantly, after which work routes back to people exactly as before. It should be a control the team already knows how to use: disabling a helpdesk trigger, pausing two workflows. If stopping the system needs the vendor, it is not a kill switch.
How each production system keeps a human in the loop
The same four places, applied differently to each of the six jobs at a US marina-software company.
| SYSTEM | THE MACHINE DOES | THE PERSON DOES | NEVER AUTOMATED |
|---|---|---|---|
| AI support agent | Triages, answers from the docs, closes about 60% of tickets | Handles escalations that arrive with notes; samples auto-closed tickets | Refunds, account changes, deletions |
| AI dev agent | Turns a ticket into a tested pull request; retries a red build within a limit | Reviews and merges every pull request | Merging |
| AI test automation | Generates and runs tests; heals selectors and labels broken by intentional change | Reviews every heal and a sample of the runs; decides what a real failure means for the release | Healing an assertion about behaviour |
| Autonomous bug fixer | Reproduces, root-causes, fixes with tests, opens the PR | Reviews the fix and the test that proves it | Merging, closing the bug without review |
| Documentation automation | Audits the docs against the product and the support log, drafts what is missing | Approves every filed draft before it is applied | Publishing a doc change without review |
| Voice AI agent | Verifies the caller, answers from the docs, logs the ticket | Takes the transferred call the moment judgment is needed | Billing, account changes on the phone |
The same four places apply outside the AI workforce. The dealership enrichment pipeline resolves each rooftop's scheduler from several sources, a person reviews everything below the confidence gate, and a low-confidence answer is never written to the CRM.
Common mistakes
A checklist for the vendor meeting
Ask each of these and expect a specific answer:
- What is the confidence threshold, who sets it, and where do we see it?
- What is the exact allowed-action list, and how is it enforced, by permission or by prompt?
- What does an escalation contain when it reaches our person?
- Where is the audit trail, and can we read a single decision end to end?
- What is the kill switch, and can our admin operate it without you?
- Which actions are never automated, by design?
A vendor who can answer all six has built a system; one who cannot has built a demo.
Questions about human-in-the-loop automation
Does human-in-the-loop mean a person checks every output?
No. It means a person is placed at pre-decided points: below a confidence threshold, before irreversible actions, on a sample, and at a kill switch. A system in which a person checks every output has not been automated, and the checking degrades into rubber-stamping within weeks.
How is the confidence threshold set?
By the team, starting conservative and tuned weekly against the reopen and escalation rates. The agent scores each draft against the sources it found and the clarity of the request; the threshold is the score below which it does not act. It is a visible dial, not a vendor secret.
Is a prompt instruction enough to stop an agent doing something?
No. A prompt is guidance; a permission the agent does not hold is a guarantee. Irreversible actions such as refunds, account changes, deletions, and merges are kept off the agent's allowed-action list at the API and permission level, so a wrong answer can be wrong but never irreversible.
Does keeping a human in the loop cancel out the savings?
No, when the loop is placed correctly. At the marina-software client the support agent closes about 60% of tickets with no person involved (rounded from production data, 2025–2026). The rest reach a person as prepared cases: the intent, the sources it found, the draft, and the reason it stopped. The savings come from the machine owning the repeats; the loop is what makes that safe enough to run.
Related pages
Decide where your people belong, on paper, first.
The free Automation Audit maps your repetitive work and marks the four places for a person in each job. 30 minutes, written Automation Map within 48 hours.