Sign up to see the future, today<\/p>\n
Can\u2019t-miss innovations from the bleeding edge of science and tech<\/p>\n
In a matter of only a few years, AI chatbots have become a common part of many of our daily lives, even though they remain deeply flawed systems.<\/p>\n
The reality is that chatbots like OpenAI\u2019s ChatGPT, Google\u2019s Gemini, or Anthropic\u2019s Claude still make regular mistakes. According to an October study by the BBC<\/a>, even the most advanced AI chatbots gave wrong answers a whopping 45 percent of the time.<\/p>\n But many users don\u2019t understand that reality. As detailed in a new paper<\/a>, University of Pennsylvania postdoctoral researcher Steven Shaw and marketing professor Gideon Nave found that in a series of experiments, users tended to take the output of ChatGPT at face value even when it gave them the incorrect answer.<\/p>\n Across a series of experiments, participants were asked to answer a variety of reasoning and knowledge-based questions. Despite making the use of ChatGPT optional, over 50 percent of them chose to use the chatbot to answer the questions.<\/p>\n The researchers were testing a key theory: whether users would be willing to believe what the AI was telling them regardless of accuracy, in what they termed a \u201ccognitive surrender\u201d that effectively overrode their intuition and deliberation process.<\/p>\n In the most striking experiment, involving 359 participants, participants followed AI\u2019s correct advice 92.7 percent of the time \u2014 and a still-considerable 79.8 percent of the time when the AI gave them the wrong answer.<\/p>\n \u201cWhile override rates were substantially higher on AI-Faulty than AI-Accurate trials, participants followed faulty AI recommendations on roughly four out of five chat-engaged trials,\u201d the researchers wrote. <\/p>\n The research points at a much broader change in how we perceive the world around us and how we\u2019re letting AI influence how we make decisions.<\/p>\n \u201cWe felt that the ability to actually outsource thinking hadn\u2019t really been studied itself. It\u2019s sort of a profound idea,\u201d Shaw said during a UPenn podcast appearance<\/a> last month. \u201cA bit provocative, I would say, in the paper, that with these AI tools that are available, they\u2019re so ingrained in our daily lives and decision processes that we now have the option or ability to outsource thinking itself.\u201d<\/p>\n The results suggest that users are willing to give up their own agency when AI presents them with false-but-plausible directions.<\/p>\n \u201cWe saw that even when cognitive surrender is engaged, people adopt those answers and are more confident in those answers,\u201d Shaw explained during the podcast episode.<\/p>\n The experiments also suggest we could be losing our ability to critically engage with information, something previous research<\/a> has found<\/a> as well.<\/p>\n \u201cThe capacity to think critically, the capacity to be able to check what the AI is giving you has become more and more important over time,\u201d Nave said. \u201cThis is kind of a muscle that we have, that hopefully we are not going to lose over time.\u201d<\/p>\n \u201cRight now, we are constrained by communicating with LLMs through our phones or our computers,\u201d Shaw added. \u201cAs those barriers reduce, that integration is just going to become stronger.\u201d<\/p>\n Eventually, we could continue giving up our agency, further cementing our reliance on AI.<\/p>\n \u201cEverybody thinks that this point will come from AI getting better and better,\u201d Nave said. \u201cBut there is an alternative story here, of humans becoming more and more reliant on AI. Just like we now have an air conditioner that can set our temperature easily, and we can move from one place to another without using any physical activity.\u201d<\/p>\n \u201cJust like many of us have lost something because of this cultural or technological evolution, we may lose as a species something very critical to our existence,\u201d he added, \u201cwhich is our capacity to think.\u201d<\/p>\n