Routing by risk solves the throughput problem. It does not solve the harder one, which is that reviewers stop reviewing.

Washington DC

Bottom line. Do not review every output, and do not review none. Route by consequence. Auto-approve the reversible low-stakes cases and hold the irreversible high-stakes ones for a person. Put the middle tier into async review, where work proceeds under constraints while a human catches up. Sample the auto-approved tier continuously rather than trusting it. Then budget for the failure nobody plans for, which is that your reviewers will drift into approving by default, faster the more accurate your model becomes. Measure reviewer disagreement rate as a health metric. A review queue with a 2 percent rejection rate is not evidence of a good model.

Route by consequence, not by volume

Put a person in front of every model output and you have rebuilt the bottleneck automation was meant to remove. Put nobody there and you are shipping unreviewed generated content into production. Neither works. Most implementations solve it the same way.

Route by risk. Most implementations use three or four tiers.

The bottom tier auto-executes: high confidence, low consequence, reversible, with a preapproved pattern already established. No human touches it.

The top tier blocks until a person approves: high consequence, irreversible, regulated, or low model confidence. This is the tier where an approval gate belongs.

The middle tier is where the useful design work happens. Work proceeds under constraints while review happens asynchronously. A draft publishes to a limited audience, or executes with a scope limit, or ships with a disclaimer attached, and a human catches up behind it. That tier keeps the queue from becoming a stop-the-world event.

Two details separate a working system from a diagram.

Calibrate thresholds to the cost of failure rather than to average accuracy. A model that is 95 percent accurate is fine for tagging support tickets and unacceptable for anything that touches money or a regulator. The threshold is a function of what the error costs, not of how the model scores.

Give reviewers structured outcomes instead of approve and reject. Approve as-is, approve with constraints, request revision with a reason code, escalate. Binary buttons throw away the reason, and the reason is the training signal that improves the routing.

Reviewers stop reviewing

Now the part that most treatments of this skip.

Human oversight degrades. Reviewers who process high volumes of mostly-correct AI output stop scrutinizing it and start approving by default. The audit log keeps recording a human decision.

The name for this is automation bias. It has two halves, and both matter.

The first is over-reliance on what an automated system produces, which shows up as accepting an output without independently checking it. Researchers call these errors of commission: the system suggested something wrong and the person went along with it.

The second half gets less attention and causes more damage. Automation bias also means discounting evidence that contradicts the machine. A reviewer holding information the system does not have will tend to defer anyway, and will sometimes stop looking for contradicting information at all. Those are errors of omission: the person had what they needed to catch the mistake and did not use it.

The term comes from aviation human factors research in the 1990s, where researchers studying cockpit automation found pilots following flight management systems into errors they would have caught unaided. Parasuraman and Manzey’s 2010 review extended it across domains, and Skitka and colleagues documented it experimentally in 1999. None of that research involved AI. It described what happens to people monitoring any reliable automated system, and language models are a new instance of an old pattern rather than a new phenomenon.

A Harvard Business School field experiment measured it directly. Jacqueline Lane, Léonard Boussioux, and colleagues at the University of Washington and MIT Solve ran 228 evaluators through 3,002 screening decisions on 48 real submissions to a global health challenge. Three conditions: human only, an AI giving bare recommendations, and an AI giving recommendations with written rationales.

Both AI conditions increased how often evaluators went along with the recommendation. The narrative condition, where the model explained its reasoning, increased it more.

The mechanism the authors identify is the part to sit with. Narrative explanations suppress what they call productive overrides, meaning the cases where an evaluator catches a flawed AI judgment and corrects it. The explanations discourage independent verification, and fluent rationales end up replacing evaluation of the underlying submission. Under cognitive load, evaluators process the narrative through a persuasion route rather than an analytical one, treating surface coherence as a proxy for correctness.

Better explainability produced worse oversight. The explanation did not equip the reviewer to evaluate the recommendation. It made the recommendation easier to accept.

One further finding matters for anyone screening ideas rather than transactions. The deference was asymmetric: evaluators followed the AI’s rejection recommendations more readily than its acceptances. The authors note this may have systematically filtered out moonshot ideas, the novel approaches that do not fit conventional evaluation criteria. AI assistance raised the average quality of what got selected and may have removed the outliers. Rory Sutherland warns about the same tradeoff when he argues that innovation happens at the extremes rather than the middle.

None of this is limited to novices or low-stakes work. Automation bias has been documented across aviation, radiology, criminal justice, and hiring.

A 2025 technical brief from the European Data Protection Supervisor names the consequence. If a human operator accepts a system’s output as valid mainly because an automated system produced it, the process becomes equivalent to an automated one. The human is present, the paperwork is complete, and no oversight occurred.

Regulators have noticed. The EU AI Act requires human oversight for high-risk systems under Article 14. Article 14(4b) obliges providers to enable overseers to stay aware of their own tendency to over-rely on system output. Automation bias is the only psychological bias the Act names.

Better models make oversight worse

Oversight quality decays as model accuracy rises.

A model that is wrong often keeps its reviewers sharp, since they catch errors regularly and stay engaged. A model that is right 99 percent of the time trains its reviewers to approve. The vigilance required goes up as the perceived need for it goes down. Humans are poor at sustained monitoring of highly reliable systems, and that finding predates AI by decades.

Which produces an uncomfortable operational fact. The better your model performs, the more your oversight layer costs to maintain and the less it appears to be worth. A review step with a 2 percent rejection rate looks like proof the model is good. It is equally consistent with reviewers who stopped reading.

You cannot distinguish those two cases from the rejection rate alone, which is why the rejection rate is a bad health metric on its own.

Add friction where agreement is too easy

The mechanisms that work introduce friction deliberately, at the point where agreement is too easy.

Divergence detection. Run two independent checks and route to a human only when they disagree. The disagreement itself is the signal, and it saves the reviewer’s attention for cases where attention has something to bite on. Some implementations run the same model twice and use the consistency of the two answers as an uncertainty estimate. Only the inconsistent outputs go to a human.

Calibrated abstention. Build the model to decline rather than guess when it is uncertain. An abstention is a routing decision the model makes for you, and it is more reliable than a confidence score you threshold after the fact.

Continuous sampling of the auto-approved tier. Pull a random or risk-weighted sample of the outputs nobody reviewed and review them. This is acceptance sampling, which manufacturing has used for a century. It detects drift, and it keeps reviewers engaged with a stream that includes real errors.

Seeded errors. Some quality programs inject known-bad outputs into the review queue at a low rate. If reviewers approve the seeded errors, you have measured your rubber-stamp rate directly rather than inferring it. This is uncomfortable to propose and it is the only direct measurement available.

Withhold the explanation until after the judgment. Given the Harvard finding, showing a reviewer the model’s reasoning before they form their own view increases deference. Ask for the reviewer’s assessment first, then reveal what the model concluded and why. The order costs nothing and it preserves independent judgment.

Cap the queue. Automation bias worsens under time pressure and cognitive load. A reviewer processing four hundred items in a shift is not performing the same task as one processing forty, whatever the job description says. If the queue depth requires speed, you have designed a rubber stamp with extra steps.

What the vendors are recommending

The companies building these systems recommend the tiering described above, plus one addition.

Anthropic’s 2026 Agentic Coding Trends Report lists scaling human-agent oversight among four priorities for the year. They describe the shift as moving from reviewing everything to reviewing what matters. Their framing has three parts. Agentic quality control, where AI agents review large-scale AI output for security and consistency problems at a volume that would overwhelm people. Agents that recognize when to ask for help, flagging their own uncertainty and escalating decisions with business impact. And human attention concentrated where it changes the outcome.

The first part is the addition. Using models to perform the first-pass review of model output is now standard practice rather than a novelty, and it makes the volume tractable. It also introduces a dependency. Your review layer now has the same failure modes as the thing it reviews, which argues for keeping the sampling audit human.

Anthropic’s separate research on agent autonomy reaches a conclusion that fits the argument here. Effective oversight will require new post-deployment monitoring infrastructure alongside new interaction patterns, rather than an approval button. Their headline finding is that most agent actions are low-risk and reversible with humans usually involved. They immediately caveat it. Those averages obscure what is happening at the frontier. An average across a deployment population tells you little about the risky tail, which is the same problem averages always have.

What to measure

Reviewer disagreement rate, tracked over time per reviewer. A rate trending toward zero is the signal that oversight is decaying, and it is visible long before an incident.

Time spent per review, distributed rather than averaged. A median of eleven seconds tells you something the mean will hide.

Seeded error catch rate, if you are willing to run it. This is the only number that measures oversight directly rather than by proxy.

Sample audit findings from the auto-approved tier, as a check on whether your routing thresholds are still correct. Model behavior drifts, and a threshold set in March is a guess by September.

Build the routing, then instrument the humans

Human oversight of AI output is not a solved problem, and the treatments that present a tiered routing diagram as the answer are describing the easy half.

The easy half is throughput, and routing by consequence handles it. The hard half is that the oversight layer degrades invisibly, accelerates as the model improves, and produces an audit trail indistinguishable from real review.

Build the routing. Then build the instrumentation that tells you whether the humans in your loop are still doing anything. The default outcome is that they quietly stop, and the paperwork will not tell you.