The output looks right. It is well written, confident, specific, and it arrives in seconds. So the human skims it, nods, and ships it. Days later someone notices the citation was invented, the number was wrong, the logic quietly broke halfway through. Nobody was careless in the usual sense. They were trusting.
This is the confidence trap, and it is the failure mode that matters most as AI spreads through knowledge work. Models are not dangerous mainly because they are sometimes wrong. They are dangerous because they are wrong in the same fluent, assured voice they use when they are right, and that voice switches off the human habit of checking.
The jagged frontier
The best evidence for this comes from the same trial that showed AI's benefits. In a randomized study of 758 BCG consultants, GPT-4 users on suitable tasks did more work, faster, at higher quality. But the researchers also gave consultants a task built to fall just outside the model's real competence, a problem that looked like the others but required judgment the model didn't have. On that task, the consultants using AI performed worse than those without it. They accepted plausible, confident answers that were wrong.
The researchers named the underlying shape the jagged frontier: AI is startlingly capable on one task and quietly incompetent on a neighboring one that looks identical from the outside. The boundary is invisible. And because the model sounds equally sure on both sides of it, the human has no built-in signal for when to stop trusting. The failure the study described was not the model's error. It was the human's decision to stop steering, sometimes called falling asleep at the wheel.
Why people want the wheel back
The workers themselves sense this. In a large audit of what people actually want from AI agents, across 1,500 workers and hundreds of tasks, the preference was clear: automate the low-value drudgery, but keep shared agency on high-value work, where the AI can execute but the human retains judgment and responsibility. People are not asking to be replaced by a confident narrator. They are asking to stay accountable for the parts that matter, which is exactly where the confidence trap does the most damage.
Judgment is a muscle you have to use
As Ricardo Semler has told it, when he collapsed from overwork at twenty-five, a doctor called him the healthiest sick person he had ever seen, and that afternoon Semler began measuring the company differently: not whether people had put in the hours, but whether the pump had actually shipped. The point was to look at the real result rather than the reassuring proxy.
The confidence trap is a proxy problem. Fluent, confident prose is a proxy for correctness, and a bad one. The discipline that defeats the trap is the same Semler reached for: check the thing itself. Did the number reconcile? Does the citation exist? Would this hold up if a skeptical colleague read it line by line? Treating the model's output as a claim to be verified, not an answer to be accepted, is the single habit that separates people who benefit from AI from people it quietly embarrasses.
Fluency is not truth
The organizations that use AI well build the check into the workflow rather than hoping for vigilance. High-stakes output gets a second set of eyes. Anything the model asserts as fact gets traced to a source. Teams talk openly about where the model has burned them, so the map of the jagged frontier is shared rather than relearned by each person the hard way.
The trap is worse where it is least expected: on work that looks routine. A wildly complex request puts a person on guard, so they check. A request that resembles a hundred the model has handled well invites the skim, which is exactly when the jagged frontier bites. The habit that protects against it is to calibrate scrutiny to stakes rather than to apparent difficulty, and to reserve the deepest checking for the answers that arrive most smoothly.
The trap will not go away as models improve; better models fail at a higher, more convincing level, which makes the confident error harder to catch, not easier. The defense is cultural and permanent: remember that fluency is not truth, that confidence is not competence, and that the human at the wheel is still the one responsible when the plausible answer turns out to be wrong.
Sources
- Dell'Acqua, McFowland, Mollick, Lifshitz-Assaf, Kellogg, Lakhani et al. (2025). Navigating the Jagged Technological Frontier (Organization Science). RCT with 758 BCG consultants: on tasks inside AI's frontier, GPT-4 users did 12.2% more tasks, 25.1% faster, ~40% higher quality; on tasks outside it they erred more. https://pubsonline.informs.org/doi/10.1287/orsc.2025.21838
- Shao, Zope, Jiang, Pei, Brynjolfsson, Yang et al. (2025). Future of Work with AI Agents (arXiv 2506.06576). Across 1,500 workers, 844 tasks and 104 occupations, people want AI to automate low-value tasks but prefer shared agency (AI executes, human keeps judgment) on high-value work. https://arxiv.org/abs/2506.06576