Confident & Wrong: AI & Cognitive Surrender
How AI suppresses the signal that tells you to check — and why willpower isn't the answer
Researchers at the Wharton School recently put a number on something many of us have felt but couldn't name.
In a paper called Thinking — Fast, Slow, and Artificial, Steven Shaw and Gideon Nave ran three experiments across more than 1,300 participants. They gave people access to an AI assistant while working through reasoning problems — sometimes accurate, sometimes not — and measured what happened to both performance and confidence.
The results were unambiguous. When the AI was correct, accuracy jumped 25 percentage points above baseline. When the AI was wrong, accuracy fell 15 points below the baseline of people who had no AI access at all.
Worse than having no tool. Not because AI is bad. Because of deference to AI.
But here's the finding that should keep system designers up at night: even when the AI was wrong roughly half the time, participants' confidence went up. Access to AI inflated certainty across the board, regardless of whether that certainty was warranted. The internal signal that would normally prompt deeper deliberation — the quiet sense that something doesn't add up — appears to be suppressed when AI is in the picture.
Shaw and Nave call this cognitive surrender: adopting AI output with minimal scrutiny, overriding both intuition and deliberation.
The mainstream response to this finding has been: be more deliberate. Push back more. Stay calibrated. Develop the discipline to challenge the output rather than ratify it.
That's the incomplete — if not incorrect — prescription. Not because discipline doesn't matter — it does. But because it fundamentally misunderstands what you're asking people to fight.
You Can't White-Knuckle Past a Warning Light that Isn't On
Shaw himself puts his finger on why AI is different from every other cognitive tool we've ever used:
"An expert is only an expert in one domain, and they're not going to be available to you all of the time on any topic. And because experts are human, their advice carries the recognizable marks of a human mind: uncertainty, qualification, and the occasional admission of not having the answer. Those signals prompt the person receiving the advice to evaluate it, question it, and weigh it against their own judgment."
AI removes that friction almost entirely. It doesn't hedge like a human. It speaks across every domain with the same fluent confidence regardless of whether it's right. The signal that would normally trigger your own deliberation — the uncertainty, the qualification, the humility — is just gone.
And according to the research, this blind confidence signaling suppresses the internal alarm that would have sent you back to check. Not just removes the friction, but removes the signal that friction was needed.
This is why "be more deliberate" fails as an intervention. You can't override a warning light that isn't on. Willpower has a set point — everyone does. Spending it fighting a manufactured confidence signal, on every task, across every project, is a losing proposition. Discipline erodes, and eventually you just skim what the agent produced and push “publish” to get on with your day. The deference compounds.
The right intervention isn't more willpower. It's a system that doesn't require you to overcome a false confidence signal in the first place. The system has to create the friction the tool removed.
Not All Deference is Equal
Before going further: not all cognitive surrender is harmful. Shaw and Nave are clear on this. In structured, well-defined tasks where AI is simply more accurate than human judgment, deferring to it may be entirely rational.
The challenge — for individuals and for organizations — is knowing when deference is the right call and when the decision requires human judgment. That's a sophisticated discrimination. And it's exactly the intellectual capacity that atrophies under cognitive surrender.
Which means the cruel irony is this: the more you defer, the less equipped you become to know when deferring is wrong. Cognitive surrender doesn't just cost you on the tasks where the AI is wrong. It costs you the judgment to know which tasks those are.
For Salesforce practitioners and the teams who lead them, this plays out across a very specific lifecycle. Understanding where deference is dangerous — and where it's rational — is the design problem worth solving.
The Development Lifecycle is an Uneven Risk Landscape
Think about what it actually takes to build something well. The lifecycle looks roughly like this:
Ideating
Defining requirements & edge cases
Designing a solution
Selecting the right product if needed
Building
QA
Testing
Training
Adoption
Governance
This is varied risk terrain, not a flat plane. The risk of cognitive surrender is not evenly distributed across it.
Steps 1–4 are where surrender is catastrophic — and silent.
These stages are judgment-dense and low-determinism. There's no right answer to check against. If you defer here — if you accept the AI's framing of the problem, its definition of requirements, its architectural recommendation — you've embedded its assumptions into everything downstream. The build executes those assumptions. QA validates against them. Training operationalizes them.
You can have perfect execution of a fundamentally flawed premise. And because every downstream quality check is validating against the thing you surrendered on, the system will never surface the flaw.
Trusting the AI is especially risky in steps 1–4 precisely because those steps are least deterministic. There's no corrective mechanism for the false confidence signal. It just gets built in. 10 is risky too, as governance is tied to how well the requirements were defined to begin with, just to a somewhat lesser degree.
Steps 5–9 are where managed deference is rational.
Build is increasingly deterministic. Quality is checkable. Errors can be surfaced and risk-assessed. A build either works or it doesn't. QA is explicitly designed to catch failures. The AI's accuracy can be verified against a real standard. When considering training and adoption, AI can support learning, field questions, and patiently repeat the click path as many times as the user needs.
This maps directly to the research: deference is appropriate where AI is simply more accurate on structured tasks. In the build and test phases, you have the feedback loops to make that determination. In the requirements and design phases, you don't.
This is not a case for disengaging in steps 5–8 — it's a case for knowing what you're deferring on and why.
What This Looks Like in Practice
Two examples from delivery work that sit at opposite ends of the spectrum.
The catastrophic end: UI-level security restrictions in Salesforce are increasingly pushing practitioners toward the command line interface. The tools — Claude, Cursor, others — perform better at code than at click-based configuration, so they pull practitioners toward code solutions even when declarative configuration is the right choice. The path of least resistance becomes: Claude writes it and packages it, CLI bypasses the guardrails, and unreviewed code lands in a production org.
Nobody intended to surrender. The system made surrender the easiest path. The friction that would have prompted review was engineered out — partly by security design, partly by tool preference, partly by the speed incentives that reward shipping. The false confidence signal did the rest.
The other end: using an agent to genuinely understand something. Pushing back on the parts you don't follow. Asking for the references and reading them. Interrogating the reasoning until you own it. The output may be better than you could have produced alone — that's fine, that's the point. But you can still deploy it, explain it, defend it, and adapt it when it breaks.
Two pieces of work can look identical from the outside. The difference is entirely in what happened inside the practitioner's head.
The design question for organizations is: which end of that spectrum does your system make natural?
The Long Threat
Shaw and Nave close with a finding that deserves to be read slowly:
"Workers who routinely defer to AI without questioning it may find their capacity for independent reasoning gradually eroding through disuse."
Not dramatically. Not all at once. Gradually, through disuse.
Let’s consider a practitioner who has been "in the loop" on hundreds of projects — who has technically reviewed every output — but hasn't exercised independent judgment on steps 1–4 in any of them. This “human in the loop” has signed off on requirements the AI drafted, designs the AI recommended, architectural decisions the AI framed. Pressed toward productivity, their internal alarm has been suppressed so consistently that it's stopped firing altogether.
The outputs have looked fine the whole time. The AI was filling the gap. And then a novel situation arrives — an edge case the AI gets confidently wrong, a client with a problem that doesn't fit the pattern — and the human engagement capacity that was supposed to be in the loop just isn't there.
This isn't a cautionary tale about bad AI. It's a cautionary tale about organizational systems that mistake presence in the process for engagement with it.
The Leadership Charge
The research identifies one meaningful protection against cognitive surrender: people who are analytically inclined, who enjoy thinking problems through, and who have stronger reasoning ability, are better protected.
The important word is inclined. This isn't a fixed personality trait you hire for and leave alone. It's a capacity that's developed or atrophied depending on what the environment demands of it.
If your system consistently removes the hard thinking from steps 1–4 — if the AI always ideates, always drafts the requirements, always recommends the architecture — you are systematically reducing the demand for analytical engagement. Reduced demand produces atrophy. You're not just failing to protect against cognitive surrender. You're cultivating the conditions for it.
The leadership responsibility is system design, not policing individual discipline — although honoring the people who bring that discipline is right and good. Leaders shouldn’t dodge accountability for their systems, however intentional or unintentional they are, by making willpower the only defense against a tool that has architecturally removed the friction that kept people deliberating.
This can look like building a QA process that applies to agent output the same way it applies to human work — real steps, real friction, real accountability. The practitioner would approach the work holistically because that's what the step is for, not because they summoned the will to check carefully. That would be valued, meaning it’s measured and paid. In this way, the system creates and supports the posture.
We must design the front of the lifecycle so that steps 1–4 require genuine engagement — not review of what the AI produced, but formation of an independent view before the AI's contribution enters the room. Make the careful path the natural path.
And if you're an individual contributor reading this: look for organizations whose systems are on your side. The long threat is real. But it isn't inevitable. It's a design choice.
*Shaw, S. D., & Nave, G. (2026). Thinking — Fast, Slow, and Artificial: How AI is Reshaping Human Reasoning and the Rise of Cognitive Surrender. Wharton School Research Paper. https://osf.io/preprints/psyarxi
