AI advice made people less accurate but more confident, study finds

A new study from three European universities has produced a stark data point on how AI advice degrades human judgment. Researchers found that access to an AI model collapsed participants’ willingness to say “I don’t know” from 44% to 3%. Accuracy dropped from 27% to 9%. Confidence, meanwhile, rose from 30% to 76%. The findings, published in The Next Web, add experimental weight to a phenomenon researchers at Wharton earlier this year called “cognitive surrender.”

The study, authored by Valerio Capraro of the University of Milano-Bicocca, Chiara Marcoccia of École Normale Supérieure, and Walter Quattrociocchi of Sapienza University of Rome, deliberately used questions where AI models typically fail. Participants were asked about visual details from films, such as the colour of a team’s uniform in Bend It Like Beckham. The researchers used Google’s Step 3.5 Flash, a model that was usually wrong on these questions. This design choice matters. It ensures that any reduction in judgment cannot be explained as sensible delegation to a reliable tool. Some participants who would have answered correctly on their own asked the AI and became wrong.

“People became much worse, the accuracy was only one third, but they were twice as confident,” Capraro said.

The numbers tell a clear story. Without AI, 44% of participants recognized the limits of their knowledge and declined to answer. With AI access, that figure collapsed to 3%. The AI was not just providing wrong answers. It was suppressing the cognitive habit of recognizing what you do not know. The mechanism appears to be the availability of an answer, not the correctness of it. The AI’s presence alone shifted behavior.

Monetary incentives helped, but not much. When participants were paid for correct answers, willingness to admit ignorance rose from 3% to 8% and accuracy from 9% to 16%. Both figures remained well below the no-AI baselines of 44% and 27%. Even financial stakes could not restore the judgment that the AI had displaced.

Wharton researchers earlier this year coined the term “cognitive surrender” to describe the same phenomenon. Their work found people accepting incorrect AI answers 80% of the time while reporting higher confidence than those working without AI. The new study adds a sharper data point. It is not just that people trust wrong AI answers. It is that the mere availability of AI suppresses the cognitive habit of recognizing what you do not know.

Capraro expressed particular concern about children. “For humans, the capacity to say ‘I don’t know’ is very important because it represents the recognition of the limits of our own knowledge,” he said. Children are growing up with these systems before they have developed critical thinking skills. Google’s AI search overhaul replaced links with confident AI-generated summaries. Common Sense Media this week called that design an “unacceptable risk” for students.

The pattern is consistent. AI products are designed to answer, never to say “I don’t know.” The humans using them are learning to do the same.

This is not a problem that better models will solve. The study used a deliberately weak model to isolate the effect of AI availability from the effect of AI reliability. But even with a perfect model, the dynamic would be different only in degree. The cognitive habit of recognizing ignorance would still be suppressed by the presence of an answer. The question is whether AI systems can be designed to preserve that habit.

Some AI companies have experimented with uncertainty signaling. OpenAI’s GPT-4o and Anthropic’s Claude 3.5 Sonnet include confidence estimates or refusal mechanisms for uncertain answers. But these features are optional and rarely the default. The default for every major AI product is to answer. The commercial incentive is to appear helpful, not to say “I don’t know.”

The study suggests a design principle that is almost never followed. AI systems should be optimized not just for accuracy but for preserving human judgment. That means sometimes refusing to answer. That means signaling uncertainty clearly. That means designing interfaces that encourage users to think, not just to accept.

The outstanding question is whether the market will reward such design. The companies that win on user engagement are the ones that answer fastest and most confidently. The companies that suppress cognitive surrender may lose users to the ones that do not. That is the tension at the heart of the AI product economy.