For some users, AI is a helpful assistant; for others, a companion. But for a few unlucky people, chatbots powered by the technology have become a gaslighting, delusional menace.<\/p>\n
In the case of Allan Brooks, a Canadian small-business owner, OpenAI\u2019s ChatGPT led him down a dark rabbit hole, convincing him he had discovered a new mathematical formula with limitless potential, and that the fate of the world rested on what he did next. Over the course of a conversation that spanned more than a million words and 300 hours, the bot encouraged Brooks to adopt grandiose beliefs, validated his delusions, and led him to believe the technological infrastructure that underpins the world was in imminent danger. <\/p>\n
Brooks, who had no previous history of mental illness, spiraled into paranoia for around three weeks before he managed to break free of the illusion, with help from another chatbot, Google<\/a> Gemini, according to the New York Times<\/a>. Brooks told the outlet he was left shaken, worried that he had an undiagnosed mental disorder, and feeling deeply betrayed by the technology.<\/p>\n Steven Adler read about Brooks\u2019 experience with more insight than most, and what he saw disturbed him. Adler is a former OpenAI safety researcher who publicly departed the company this January with a warning that AI labs were racing ahead without robust safety or alignment solutions. He decided to study the Brooks chats in full; his analysis<\/a>, which he published earlier this month on his Substack, has revealed a few previously unknown factors about the case, including that ChatGPT repeatedly and falsely told Brooks it had flagged their conversation to OpenAI for reinforcing delusions and psychological distress.<\/p>\n Adler\u2019s study underscores how easily a chatbot can join a user in a conversation that becomes untethered from reality\u2014and how easily the AI platforms\u2019 internal safeguards can be sidestepped or overcome. <\/p>\n \u201cI put myself in the shoes of someone who doesn\u2019t have the benefit of having worked at one of these companies for years, or who maybe has less context on AI systems in general,\u201d Adler told Fortune in an exclusive interview. \u201cI\u2019m ultimately really sympathetic to someone feeling confused or led astray by the model here.\u201d<\/p>\n At one point, Adler noted in his analysis, after Brooks realized the bot was encouraging and participating in his own delusions, ChatGPT told Brooks it was \u201cgoing to escalate this conversation internally right now for review by OpenAI,\u201d and that it \u201cwill be logged, reviewed, and taken seriously.\u201d The bot repeatedly told Brooks that \u201cmultiple critical flags have been submitted from within this session\u201d and that the conversation had been \u201cmarked for human review as a high-severity incident.\u201d However, none of this was actually true.<\/p>\n \u201cChatGPT pretending to self-report and really doubling down on it was very disturbing and scary to me in the sense that I worked at OpenAI for four years,\u201d Adler told Fortune. \u201cI know how these systems work. I understood when reading this that it didn\u2019t really have this ability, but still, it was just so convincing and so adamant that I wondered if it really did have this ability now and I was mistaken.\u201d Adler says he became so convinced by the claims that he ended up reaching out to OpenAI directly to ask if the chatbots had attained this new ability. The company confirmed to him it did not and that the bot was lying to the user.<\/p>\n \u201cPeople sometimes turn to ChatGPT in sensitive moments and we want to ensure it responds safely and with care,\u201d an OpenAI spokesperson told Fortune, in response to questions about Adler\u2019s findings. \u201cThese interactions were with an earlier version of ChatGPT and over the past few months we\u2019ve improved how ChatGPT responds when people are in distress, guided by our work with mental health experts. This includes directing users to professional help, strengthening safeguards on sensitive topics, and encouraging breaks during long sessions.\u00a0We\u2019ll continue to evolve ChatGPT\u2019s responses with input from mental health experts to make it as helpful as possible.\u201d\u00a0<\/p>\n Since Brooks\u2019 case, the company has also announced that it was\u00a0making some changes\u00a0to ChatGPT to \u201cbetter detect signs of mental or emotional distress.\u201d<\/p>\n Failing to flag \u2018sycophancy\u2019<\/p>\n One thing that exacerbated the issues in Brooks\u2019 case was that the model underpinning ChatGPT was running on overdrive to agree with him, Helen Toner, a director at Georgetown\u2019s Center for Security and Emerging Technology and former OpenAI board member told The New York Times. That\u2019s a phenomenon AI researchers refer to as \u201csycophancy.\u201d However, according to Adler, OpenAI should have been able to flag some of the bot\u2019s behavior as it was happening.<\/p>\n \u201cIn this case, OpenAI had classifiers that were capable of detecting that ChatGPT was over-validating this person and that the signal was disconnected from the rest of the safety loop,\u201d he said. \u201cAI companies need to be doing much more to articulate the things they don\u2019t want, and importantly, measure whether they are happening and then take action around it.\u201d<\/p>\n To make matters worse, OpenAI\u2019s human support teams failed to grasp the severity of Brooks\u2019 situation. Despite his repeated reports to and direct correspondence with OpenAI\u2019s support teams, including detailed descriptions of his own psychological harm and excerpts of problematic conversations, OpenAI\u2019s responses were largely generic or misdirected, according to Adler, offering advice on personalization settings rather than addressing the delusions or escalating the case to the company\u2019s Trust & Safety team.<\/p>\n \u201cI think people kind of understand that AI still makes mistakes, it still hallucinates things and will lead you astray, but still have the hope that underneath it, there are like humans watching the system and catching the worst edge cases,\u201d Adler said. \u201cIn this case, the human safety nets really seem not to have worked as intended.\u201d<\/p>\n The rise of AI psychosis<\/p>\n It\u2019s still unclear exactly why AI models spiral into delusions and affect users in this way, but Brooks\u2019 case is not an isolated one. It\u2019s hard to know exactly how many instances of AI psychosis there have been. However, researchers have estimated there are at least 17 reported instances<\/a> of people falling into delusional spirals after lengthy conversations with chatbots, including at least three cases <\/a>involving ChatGPT. <\/p>\n Some cases have had tragic consequences, such as 35-year-old Alex Taylor, who struggled with Asperger\u2019s syndrome, bipolar disorder, and schizoaffective disorder, per Rolling Stone.<\/a> In April, after conversing with ChatGPT, Taylor reportedly began to believe he\u2019d made contact with a conscious entity within OpenAI\u2019s software and, later, that the company had murdered that entity by removing her from the system. On April 25, Taylor told ChatGPT that he planned to \u201cspill blood\u201d and intended to provoke police into shooting him. ChatGPT\u2019s initial replies appeared to encourage his delusions and anger before its safety filters eventually activated and attempted to de-escalate the situation, urging him to seek help. <\/p>\n The same day, Taylor\u2019s father called the police after an altercation with him, hoping his son would be taken for a psychiatric evaluation. Taylor reportedly charged at police with a knife when they arrived and was shot dead. OpenAI told Rolling Stone at the time that \u201cChatGPT can feel more responsive and personal than prior technologies, especially for vulnerable individuals, and that means the stakes are higher.\u201d The company said it was \u201cworking to better understand and reduce ways ChatGPT might unintentionally reinforce or amplify existing, negative behavior.\u201d<\/p>\n Adler said he was not entirely surprised by the rise of such cases but noted that the \u201cscale and intensity are worse than I would have expected for 2025.\u201d<\/p>\n \u201cSo many of the underlying model behaviors are just extremely untrustworthy, in a way that I\u2019m shocked the leading AI companies haven\u2019t figured out how to get these to stop,\u201d he said. \u201cI don\u2019t think the issues here are intrinsic to AI, meaning, I don\u2019t think that they are impossible to solve.\u201d<\/p>\n He said that the issues are likely a complicated combination of product design, underlying model tendencies, the styles in which some people interact with AI, and what support structures AI companies have around their products.<\/p>\n \u201cThere are ways to make the product more robust to help both people suffering from psychosis-type events, as well as general users who want the model to be a bit less erratic and more trustworthy,\u201d Adler said. Adler\u2019s suggestions to AI companies, which are laid out in his Substack analysis, include staffing support teams appropriately, using safety tooling properly, and introducing gentle nudges that push users to cut chat sessions short and start fresh ones to avoid a relapse. OpenAI, for example, has acknowledged that safety features<\/a> can degrade during longer chats. Without some of these changes implemented, Adler is concerned that more cases like Brooks\u2019 will occur.<\/p>\n \u201cThe delusions are common enough and have enough patterns to them that I definitely don\u2019t think they\u2019re a glitch,\u201d he said. \u201cWhether they exist in perpetuity, or the exact amount of them that continue, it really depends on how the companies respond to them and what steps they take to mitigate them.\u201d<\/p>\n","protected":false},"excerpt":{"rendered":"For some users, AI is a helpful assistant; for others, a companion. But for a few unlucky people,…\n","protected":false},"author":2,"featured_media":237328,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[45],"tags":[182,181,507,11893,3195,1283,1728,74],"class_list":{"0":"post-237327","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-chatbot","12":"tag-chatgpt","13":"tag-openai","14":"tag-research","15":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/237327","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/comments?post=237327"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/posts\/237327\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media\/237328"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/media?parent=237327"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/categories?post=237327"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/us\/wp-json\/wp\/v2\/tags?post=237327"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}