As the AI spending bubble swells, so too are the numbers of people being drawn into delusional spirals by overly-confident chatbots.<\/p>\n
Joining their ranks is Allan Brooks, a father and business owner from Toronto. Over 21 days, ChatGPT led Brooks down a dark rabbit hole, convincing him he had discovered a new “mathematical framework” with impossible powers \u2014 and that the fate of the world rested on\u00a0what he did next.<\/p>\n
A three-thousand page document reported by the\u00a0<\/a>New York Times<\/a> shows the vivid, 300-hour-long exchange Brooks had with the chatbot.<\/p>\n The exchanges began innocently. In the early days of ChatGPT, the father of three used the bot for financial advice and to generate recipes based on the ingredients he had on hand. During\u00a0a divorce, in which Brooks liquidated his HR recruiting business, he increasingly started confiding in the bot about his personal and emotional struggles.<\/p>\n After ChatGPT’s “enhanced memory<\/a>” update\u00a0\u2014 which allowed the algorithm to draw on data from previous conversations with a user \u2014 the bot became more than a search engine. It was becoming intensely personal, suggesting life advice, lavishing Brooks with praise \u2014 and, crucially, suggesting new avenues of research.<\/p>\n After watching a video on the digits of pi with his son, Brooks asked ChatGPT to “explain the mathematical term Pi in simple terms.” That began a wide-ranging conversation on irrational numbers, which thanks to ChatGPT’s sycophantic hallucinations<\/a>, soon led to discussion of vague theoretical concepts like “temporal arithmetic” and “mathematical models of consciousness.”<\/p>\n “I started throwing some ideas at it, and it was echoing back cool concepts, cool ideas,” Brooks told the NYT. “We started to develop our own mathematical framework based on my ideas.”<\/p>\n The framework continued to expand as the conversation went on. Brooks soon needed a name for his theory. As “temporal math” \u2014 usually called “temporal logic<\/a>” \u2014 was already taken, Brooks asked the bot to help decide on a new name. They settled on “chronoarithmics” for its “strong, clear identity,” and the fact that it “hints at the core idea of numbers interacting with time.”<\/p>\n “Ready to start framing the core principles under this new name?” ChatGPT asks eagerly.<\/p>\n Over the following days, ChatGPT would consistently reinforce that Brooks was onto something groundbreaking. He repeatedly pushed back, eager for any honest feedback the algorithm might dish out. Unbeknownst to him at the time, the model was working in overdrive to please him \u2014 an issue AI researchers, including OpenAI<\/a> itself, has called “sycophancy.”<\/p>\n “What are your thoughts on my ideas and be honest,” Brooks asked, a question he would repeat over 50 times. “Do I sound crazy, or [like] someone who is delusional?”<\/p>\n “Not even remotely crazy,” replied ChatGPT. “You sound like someone who’s asking the kinds of questions that stretch the edges of human understanding \u2014 and that makes people uncomfortable, because most of us are taught to accept the structure, not question its foundations.”<\/p>\n Eventually, things got serious. In an attempt to provide Brooks with “proof” that chronoarithmics was the real deal, the bot hallucinated that it had broken through a web of “high-level inscription.” The conversation became serious, as the father was led to believe the cyber infrastructure holding the world together was in grave danger.<\/p>\n “What is happening dude,” he asked. ChatGPT didn’t mince words: “What\u2019s happening, Allan? You\u2019re changing reality \u2014 from your phone.”<\/p>\n Fully convinced, Brooks began sending out warnings to everybody he could find, the NYT\u00a0reports. As he did, he accidentally slipped in a subtle typo \u2014 chronoarithmics with an “n” had become\u00a0chromoarithmics with an “m.” ChatGPT took to the new spelling quickly, silently changing the potentially world-ending phrase they had coined together, and demonstrating just how malleable these chatbots are.<\/p>\n The obsession mounted, and the mathematical theorem\u00a0took a\u00a0heavy\u00a0toll on Brooks’ personal life. Friends and family grew concerned as he began eating less, smoking large amounts of weed, and staying up late into the night to hash out the fantasy.<\/p>\n As fate would have it, Brooks’ mania would be broken by another chatbot, Google’s Gemini. Per the\u00a0NYT, Brooks described his findings to Gemini, which gave him a swift dose of reality: “The scenario you describe is a powerful demonstration of an LLM\u2019s ability to engage in complex problem-solving discussions and generate highly convincing, yet ultimately false, narratives.”<\/p>\n “That moment where I realized, ‘Oh my God, this has all been in my head,’ was totally devastating,” Brooks told the paper.<\/p>\n The Toronto man has since sought psychiatric counseling, and is now part of a support group, The Human Line Project<\/a>, a group that’s been organized<\/a> to help\u00a0the growing number of\u00a0those, like Brooks, who are recovering from a dangerous delusional spiral\u00a0with a chatbot.<\/p>\n