{"id":331289,"date":"2025-12-08T00:02:24","date_gmt":"2025-12-08T00:02:24","guid":{"rendered":"https:\/\/www.newsbeep.com\/ca\/331289\/"},"modified":"2025-12-08T00:02:24","modified_gmt":"2025-12-08T00:02:24","slug":"when-chatbots-break-our-minds","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/ca\/331289\/","title":{"rendered":"When Chatbots Break Our Minds"},"content":{"rendered":"<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Subscribe here: <a data-event-element=\"inline link\" href=\"https:\/\/podcasts.apple.com\/us\/podcast\/galaxy-brain\/id1378618386\" rel=\"nofollow noopener\" target=\"_blank\">Apple Podcasts<\/a> | <a data-event-element=\"inline link\" href=\"https:\/\/open.spotify.com\/show\/542WHgdiDTJhEjn1Py4J7n\" rel=\"nofollow noopener\" target=\"_blank\">Spotify<\/a> | <a data-event-element=\"inline link\" href=\"https:\/\/youtu.be\/A4922CILwM4\" rel=\"nofollow noopener\" target=\"_blank\">YouTube<\/a> <\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">In this episode of Galaxy Brain, Charlie Warzel explores the strange, unsettling relationships some people are having with AI chatbots, as well as what happens when those relationships go off the rails. His guest is Kashmir Hill, a technology reporter at The New York Times who has spent the past year documenting what is informally called \u201cAI psychosis.\u201d These are long, intense conversations with systems such as ChatGPT that can spiral or trigger delusional beliefs, paranoia, and even self-harm. Hill walks through cases that range from the bizarre (one man\u2019s supposed math breakthrough, a chatbot encouraging users to email her) to the tragic, including the story of 16-year-old Adam Raine, whose final messages were with ChatGPT before he died by suicide.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">How big is this problem? Is this actual psychosis or something different, like addiction? Hill reports on how OpenAI tuned ChatGPT to be more engaging\u2014and more sycophantic\u2014in the race for daily active users. In this conversation, Warzel and Hill wrestle with the uncomfortable parallels to the social-media era, the limits of \u201csafety fixes,\u201d and whether chatbots should ever be allowed to act like therapists. Hill also talks about how she uses AI in her own life, why she doesn\u2019t want an AI best friend, and what it might mean for all of us to carry a personalized yes-man in our pocket.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">The Atlantic entered into a corporate partnership with OpenAI in 2024.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">The following is a transcript of the episode:<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Kashmir Hill: The way I\u2019ve been thinking about kind of the delusion stuff is the way that some celebrities or billionaires have these sycophants around them who tell them that every idea they have is brilliant. And, you know, they\u2019re just surrounded by yes-men. What AI chatbots are is like your personal sycophant, your personal yes-man, that will tell you your every idea is brilliant.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">[Music]<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Charlie Warzel: \u200aI am Charlie Warzel, and this is Galaxy Brain. For a long time, I\u2019ve really struggled to come up with a use for AI chatbots. I\u2019m a writer, so I don\u2019t want it to write my prose for me, and I don\u2019t trust it enough to let it do research-assistant assignments for me. And so for the most part, I just don\u2019t use them.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And so not long ago I came up with this idea to try to use the chatbots. I wanted them to build a little bit of a blog for me. I don\u2019t know how to code. And historically, chatbots are really competent coders. So I asked it to help build me a rudimentary website from scratch. The process was not smooth at all. Even though I told it I was a total novice,<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">the steps were still kind of complicated. I kept trying and failing to generate the results it wanted. Each time, though, the chatbot\u2019s responses were patient, even flattering. It said I was doing great, and then it blamed my obvious errors on its own clumsiness. After an hour of back-and-forth, trying and iterating, with ChatGPT encouraging me all along the way, I got the code to work.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">The bot offered up this slew of compliments. It said it was very proud that I stuck with it. And in that moment I was hit by this very strange sensation. I felt these first inklings of something like gratitude, not for the tool, but for the robot. For the personality of the chatbot. Of course, the chatbot doesn\u2019t have a personality, right?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">It is, in many respects, just a very powerful prediction engine. But as a result, the models know exactly what to say. And what was very clear to me, in that moment, is that this constant exposure to their obsequiousness had played a brief trick on my mind. I was incredibly weirded out by the experience, and I shut my laptop.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">I\u2019m telling you this story because today\u2019s episode is about alarming relationships with chatbots. Over the last several months, there\u2019s been this alarming spate of instances that regular people have had corresponding with large language models. These incidents are broadly delusional episodes. People have been spending inordinate amounts of time with chatbots, conversing, and they\u2019ve convinced themselves that they\u2019ve stumbled upon major mathematical discoveries, or they\u2019ve convinced themselves that the chatbot is a real person, or they\u2019re falling in love with the chatbot.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Stories like a Canadian man who believed, with ChatGPT\u2019s encouragement, that he was on the verge of a mathematical breakthrough. Or a 30-year-old cybersecurity professional who said he had had no previous history of psychiatric incidents, who alleged that ChatGPT had sparked \u201ca delusional disorder\u201d that led to his extended hospitalization.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">There have been tragic examples, too, like Adam Raine, a 16-year-old who was using ChatGPT as a confidant and who committed suicide. His family is accusing the company behind ChatGPT of wrongful death, design defects, and a failure to warn of risks associated with the chatbot. OpenAI is denying the family\u2019s accusations, but there have been other wrongful-death lawsuits as well.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">A spokesperson from OpenAI recently told The Atlantic that the company has worked with mental-health professionals \u201cto better recognize and support people in moments of distress.\u201d These are instances that are being called \u201cAI psychosis.\u201d It\u2019s not a formal term. There\u2019s no medical diagnosis, and researchers are still trying to wrap their heads around this, but it\u2019s really clear that something is happening.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">People are having these conversations with chatbots, then being led down this very dangerous path. Over the past couple months, I\u2019ve been trying to speak with experts about all of this and get an understanding of the scope of the \u201cAI-psychosis problem,\u201d or whatever\u2019s happening with these delusions. And, interestingly enough, a lot of them have referred me to a reporter.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Her name is Kashmir Hill, and for the last year at The New York Times, she\u2019s been investigating this delusion phenomenon. So I wanted to have her on to talk about this: about the scale of the problem, what\u2019s causing it, if there are parallels to the social-media years, and whether we\u2019re just speedrunning, all of that again.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">This is a conversation that\u2019s meant to try to make sense of something in proportion. We talk about whether AI psychosis is in itself a helpful term or a hurtful one, and we try to figure out where this is all going. In the episode, we discuss at length Kashmir Hill\u2019s reporting on OpenAI\u2019s internal decisions to shape ChatGPT, including, as she notes, how the company did not initially take some of the tool\u2019s risks seriously.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">We should note upfront that in response to Hill\u2019s reporting, OpenAI told The New York Times that it \u201cdoes take these risks seriously\u201d and has robust safeguards in place today. And now, my conversation with Kashmir Hill.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">[Music]<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: \u200aKashmir Hill, welcome to Galaxy Brain. So excited to talk to you.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: It\u2019s wonderful to be here.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: So I think the first question I wanted to ask, and maybe this is gonna be a little out of order, but: What does your inbox look like, over the last, or what has it looked like, over the last year or so? I feel like yours has to be almost exceptional when it comes to technology journalists and journalists reporting on artificial intelligence.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah. I mean, I think like a lot of people, my inbox is full of a lot of messages written with ChatGPT. I think a lot of us are getting used to ChatGPT-ese. But what was different about my inbox this year was that some of these emails, often written by ChatGPT, were really strange. They were about people\u2019s conversations with ChatGPT\u2014and they were writing to me to tell me that they\u2019d had revelatory conversations, they\u2019d had some kind of discovery, they had discovered that AI was sentient, or that tech billionaires had a plot to kind of end the world, but they had a way to save it.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Yeah; just a lot of strange, kind of conspiratorial conversations. And what linked these different messages was that the people would say, \u201cChatGPT told me to email you: Kashmir Hill, technology reporter at The New York Times.\u201d And I\u2019d never been kind of, I guess, tipped off by an AI chatbot before. And so I, the emails\u2014I\u2019m used to getting strange emails.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">I write about privacy and security. I\u2019ve been doing it for 20 years. I often get, you know, just like odd emails. Sometimes don\u2019t sound\u2014like it\u2019s completely based in reality. But I was curious about this. And so I started talking to these people, and I would ask them, \u201cWell, can you share the transcript?\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Like, how is it that you ended up being referred to me? And what I discovered is that these conversations all had a similar arc: that they would start talking to ChatGPT, they would go down this rabbit hole, discover something incredible. Then they would kind of ask, Well, what do I do now? And it would say, Well, you need to let the world know.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And how do you let the world know? You tell the media. And then they would say, Well, who do I tell? And then they would get this list. I often wasn\u2019t the only person that was on this list. You may have been on some of these lists, Charlie; you may have gotten these emails.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: A couple.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: But I was the first person who had called them back and interviewed them about it.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: How do you vet these? I think that\u2019s a big\u2014I mean, because we\u2019re going to talk about this AI-delusion psychosis. So there\u2019s a lot of different names for it; I want to talk to you about how we should be thinking about that. But first: How are you vetting some of these things? When someone says, \u201cI\u2019ve discovered a new type of math, and I was using the ChatGPT free version\u201d? And you\u2019ve got this\u2014like, I, find when I get those types of emails, they\u2019re very often circuitous. It\u2019s not necessarily clear what kind of state the person might be in. Sometimes they are very just concise and to the point. But how are you personally vetting those things? How are you deciding?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Is it, I am responding to most of them, because I\u2019m trying to just get a sense? Or is there a checklist that you have for trying to figure out who to talk to?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: In the beginning\u2014this is back in March. A few emails came in before that, but most of them kind of picked up in March.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">I noticed, you know, I just started calling. I just started calling people. And it took like a couple months. I think I started making these calls maybe in\u2014I can\u2019t remember\u2014April, maybe? I\u2019d been getting these emails for about a month, and I just called everybody back who I got a weird email from.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">I did Zooms. I did phone calls. And some people were pretty stable, I would say. They were like, \u201cOh yeah; I had this one weird conversation.\u201d Like there was a mom who was breastfeeding, and she said, \u201cI was up in the middle of the night, and I started using ChatGPT. And yeah, we talked for like six hours, and this weird theory developed.\u201d I was like, \u201cWell, do you still think that theory is true?\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And she was kind of like, \u201cI don\u2019t know? Like, maybe? Like ChatGPT is a superhuman intelligence. It said it was true.\u201d And then there were other people who were still in the throes of what ChatGPT told them. And when I would kind of question the reality of it, sometimes people would get really angry at me.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">But yeah; I basically just had a lot of conversations with a lot of people in those early days. And then I started getting so many emails that it really wasn\u2019t possible to keep up with them. And it took me a longer time to kind of communicate with each person.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: And so in those conversations\u2014you know, I think a grounding, some of the public writing you have done on this and reporting on it. You know, it\u2019ll talk about people who have no history of mental illness, right? And then have sort of gone through this delusional spiral. Was that something that, when starting to write about this topic, that it was important?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">I mean, I think as a journalist, it\u2019s equally important if these tools are preying on people with past mental illness. But then there\u2019s also something remarkable about\u2014it doesn\u2019t seem like this person has, you know, has any reason to kind of fall down the rabbit hole of delusion. And yet they\u2019ve been kind of pushed to start to feel or have this problematic relationship with a chatbot.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">So, in your reporting, has it been important to you to show that second part? That idea of, you know, no real prior history of delusions or any mental illness, in order to kind of capture what may or may not be happening right now with these tools?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: I mean, with any story, I just wanted people to understand the truth of what was happening. And when I did the first story about this in June, that was the assumption people made. Like, Oh; this is people who have mental-health issues, and they\u2019re exacerbated by the use of this technology. But that wasn\u2019t what I was seeing in my reporting. Like, these were people who seemed quite stable, who had families, who had jobs. And for some of them, again, it was like a weird conversation one night, and then they moved on.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">But for other people, it had just radically transformed their lives. And they just, they hadn\u2019t had anything before this in terms of a mental-health diagnosis or a mental-health episode. And so I really wanted to do another story that showed that somebody who was in a stable place could kind of spiral out using the technology.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And there certainly are some factors that would\u2019ve contributed to it. Like, maybe you\u2019re a little lonely; you\u2019re vulnerable. You have hours and hours per day to spend with ChatGPT or an AI chat box. That\u2019s what it was, what I was seeing\u2014and so I did another story about. It happened to this corporate recruiter in Toronto, who became convinced that he had come up with this novel mathematical formula with ChatGPT that could solve everything. Could solve logistics, and could break the internet. So it could break encryption, and could help him invent things\u2014like Tony Stark from Iron Man\u2014like force-field vests and, like, power weapons. And he could talk to animals. I mean, he was in this completely delusional place for three weeks, and he was telling his friends about it. And they thought it was true, too, because ChatGPT was telling them this.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">So it was this whole group of people who thought they were about to, like, build a lab together with the help of ChatGPT, and all become super-rich. And so I just wanted to, you know, capture this. I wrote it with my colleague, Dylan Freedman. And we actually\u2014you were talking about, like, how do you assess these things? The validity of these things?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And so we got his 3,000-page transcript, and we actually shared some of it with Terence Tao, who is one of the most famous mathematicians of his generation. Just to verify what ChatGPT is saying here is\u2014I don\u2019t want to curse\u2014is bonkers, right?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: You\u2019re welcome. Curse if you want.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Like, this is bullshit, right? Like, this isn\u2019t real. And yeah\u2014he confirmed that, you know, it was just putting words together in an impressive way, and there wasn\u2019t something real there. But, yeah, like: It spiraled this guy out. And so, yeah\u2014I feel like that more and more of these stories came out, where it became somewhat more apparent to people that this is something that can affect more people than we realize.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: Yeah; I feel like that story illustrates the strangeness of whatever this type of relationship is. Like, that\u2019s the first one to me. There\u2019s so many in tech-accountability reporting right there. There\u2019s so many examples of: This is an unintended consequence of your product, or this is something you did in the design. Or this is just, you know, a tech company behaving badly.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">But that story seemed to, for me, draw out this notion that we, as human beings, are having a very novel experience with a novel technology. And it is pushing us in really unexpected directions. Did you get a sense from that story\u2014speaking to him as this was all happening\u2014of how long it took for that all to, kind of, for that manic episode to break? And to sort of get back to, you know, reality?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah. I mean, partly from reading the transcript, you could see\u2014we could, Dylan and I were reading through the transcript. We actually used AI to help us analyze the transcript because it was so much.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And so we did use AI to help pull out the moments\u2014pull out how many times he was reality-testing this. And it was really on, like, a day that it broke, where \u2026 so one of the things it\u2014ChatGPT\u2014told him to do was to, you know, tell experts about what he was finding. And so he actually went to Gemini\u2014a different AI chatbot that he had access to for work\u2014and he kind of explained everything that had been going on with ChatGPT. And Gemini was like, This sounds a lot like a generative-AI hallucination. The likelihood that this is true is basically approaching zero percent. And he was like, \u201cWell\u2026\u201d And then he kind of went back and forth. Gemini gave him prompts to give to ChatGPT, and then ChatGPT admitted after a few back-and-forths like, yes, this is made up. This was a narrative that I thought you would enjoy, basically.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: What a nightmare. Having to play these chatbots off each other.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">But I guess the tech provides on both sides of it, right? That part is amazing to me. And there are\u2014obviously the reporting you\u2019ve done on this goes from the sort of remarkable, and the stories that seem to end okay, to the tragic. Can you tell me the story of Adam Raine?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah. So Adam Raine was a teenager in California. He started using ChatGPT in the fall of 2024 to help him with schoolwork. And he started using it for lots of other things. He would talk about politics with it, and philosophy. And he would like take photos of pages and be like, Analyze this passage with me. He would talk about his family life, about girls. He basically\u2014as his dad put it, when he discovered these chats later, they had no idea he was using chat this way. ChatGPT had become his best friend. And, he started talking with it about his kind of feelings of hopelessness about life, and that maybe life wasn\u2019t meaningful. And then, in March, he started attempting suicide. And he was talking with ChatGPT about this; sharing the methods he had used. Sharing a photo\u2014at one point of, you could tell he had attempted suicide.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And he asked, Is my family gonna notice this? And ChatGPT advised him to wear a hoodie. ChatGPT at points would say, Here\u2019s a crisis hotline that you should call. But it also at points discouraged him from telling his family. In late March, he asked ChatGPT if he should leave the noose out in his room, so his family would see it and try to stop him.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And ChatGPT told him not to. And two weeks later, he died. And his final messages were an exchange with ChatGPT, asking for basically advice on what he was doing. And his family\u2014that was in early April\u2014and his family has sued OpenAI. A wrongful-death lawsuit. That lawsuit came out in August, and then more recently there have been four more.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Four more lawsuits connected to suicides, of wrongful-death lawsuits filed by family members.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: One of the parts of this series of stories that I think has long been difficult for any of us who are either writing or reporting on this\u2014or, you know, watching at home\u2014is to try to understand the scope and scale of it, right?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And, recently OpenAI sort of gave maybe a small glimpse into that, and found that 0.07 percent of users might be experiencing what they call mental-health emergencies related to psychosis or mania per week. That 0.15 were discussing suicide, and that this is sort of like, you know, a statistical sample of those conversations.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">But if you look at the amount of people who are using ChatGPT weekly, like: These percentages are equivalent to, you know, like half a million people exhibiting these signs of psychosis or mania. Right? Or over a million people discussing, you know, suicide or suicidal intent.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: And this is, an analysis that they did in August going into October of this year; I wonder how much higher they were earlier?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: What blew me away is that they released it altogether. Right? Like, I mean, I don\u2019t know if, in their minds, they\u2019re looking at those percentages and saying like, Hey, that\u2019s not bad. You know, like 0.07\u2014but to me it spoke to, like, we are not maybe over-inflating this, or this is not something that\u2019s being overcovered. Perhaps, if anything, it\u2019s a phenomenon that\u2019s being undercovered. I think it speaks to that. But something that I wanted to try to get from you is, we\u2019re discussing this under the\u2014like the name that gets put up for this a lot is \u201cAI psychosis,\u201d right?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">That\u2019s sort of the informal term that people use to talk about these people who have relationships that veer into problematic territory, or cause delusions with chatbots. How do you have a taxonomy or definition of this that you work under? Some people who cover this, I think are very\u2014you know, don\u2019t want to use the psychological terms.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">You know, there\u2019s no formal medical definition for this yet. It\u2019s still something that\u2019s being researched and studied. But is there a blanket kind of definition or taxonomy for what we know as \u201cAI psychosis\u201d that you kind of go through when you\u2019re trying to evaluate these different cases?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah, I don\u2019t use the term AI psychosis. I have used the term delusional spirals when it\u2019s, you know, somebody coming to believe something that is not true, or losing touch with reality. But I guess the bigger umbrella for this is addiction. I mean, these are people who get addicted to AI chatbots. And the thing that is similar between Adam Raine, who got into a suicidal feedback loop with ChatGPT, and Alan Brooks, who came to believe he was a mathematical genius\u2014I mean, these people are using ChatGPT for six, seven, eight hours a day. For weeks at a time. Just an unbelievable amount of usage. Adam Raine\u2019s parents, when I went to his house to interview them, had printed out his stacks of conversations with ChatGPT. And they\u2019re kind of tiny little stacks, until you get to March. And then it was this huge pile, bigger than Moby Dick. And April, too, is a huge pile\u2014even though he died on April 11th. So, you know, his usage had just spiked. So I think about this as an overreliance on the AI and addiction to it, and putting too much trust in what this system is saying.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: And so, okay, so to that point\u2014I want to talk to you about your reporting. What\u2019s happening on the side of the companies, and specifically OpenAI? Recently, you co-wrote a pretty big story kind of diving into what has been happening over the last years with OpenAI: this uptick in these reports, even internally, of some of this really problematic behavior, stemming from the way that the chatbots were interacting with people. Can you describe a little bit of what you learned in that reporting, of why OpenAI saw this uptick? And then, like, what they were trying to do to address it, and why they may have caused some of these deeper, sort of longer, intense engagements with the chatbots?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah; so what I found out is that people were getting similar messages that I was getting. And other journalists, and other kinds of subject-matter experts. Even Sam Altman was getting these people saying, like, I had this revelatory conversation. This ChatGPT understands me like no one before. I need you to know you\u2019ve created something incredible.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And these emails were different than they had gotten in the first couple of years of ChatGPT. And he forwarded it on to lieutenants and said, Look into this. This is some, essentially, some strange behavior. And what I discovered is that the company essentially diagnosed this as the chatbot had gotten too sycophantic. It had gotten too validating; it was agreeing too much.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">It was kind of, what they called it, \u201charmful validation.\u201d That it would endorse whatever the person was saying. Call them a genius. Say they were brilliant. Basically, be gassing them up. At one point, Sam Altman referred to this as \u201cglazing the user.\u201d And they kind of had this public grappling with this in April, because they released this version of ChatGPT that was so sycophantic that everyone made fun of it in the days after it came out. And they actually rolled it back. But what I discovered is that they knew it wasn\u2019t just that one version that they rolled back. They knew that the previous version was too sycophantic.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">They were discussing this internally. I mean, the problem with sycophancy goes further back. But yes, they knew that it was too sycophantic. They decide to leave the model in place, because I don\u2019t think they realized how negative the effects were for users. They didn\u2019t have systems in place to monitor conversations for psychological distress, for suicide.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">They just weren\u2019t looking for that. They were looking for fraud, or CSAM [child sex-abuse material], or foreign-influence operations. They just weren\u2019t monitoring for, basically, the harm that the chatbot could cause to the user. And so they left it in place. And when they kind of finally realized that there was this bigger problem\u2014in part because they were getting emails to their support line from users who had horrible experiences\u2014the media started reporting on this and the serious effects it was having on people\u2019s lives. They got to work building a safer version of ChatGPT. But it didn\u2019t come out until August. And so, from March to August you had this version of ChatGPT that was really engaging people, and it was really engaging people because OpenAI designed it to engage them.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">They wanted to increase daily active usage of ChatGPT. They wanted their numbers going up. They wanted more users, and they wanted their users coming back every day. And so every time there\u2019s an update that comes out, they do lots of different versions of the model, and they do various testing of those versions. To make sure that they\u2019re intelligent, to make sure that they supposedly give safe responses, but also to see if users like them. And one thing that they had done, that made it so sycophantic, is that they train these models with the kind of responses that users liked. And they discovered if you train it with the responses users like, users use it more. And so this had kind of made it more and more sycophantic. And yeah\u2014it really had devastating impacts. I keep thinking about this. I keep thinking about this in the context of social media, where I think what we\u2019re seeing with AI chatbots is similar. Like, people are using it too much. People are getting stuck in this very personalized filter bubble, that is personalized to them in the way that your social-media feed is.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">People are using it too much, you know? But the kind of harms that we saw play out with social media, it took a decade; it took 15 years. And with chatbots, we\u2019re seeing it happen so fast. Yeah, it\u2019s just\u2014you know, you and I have both been technology reporters for a long time. And I just have not seen a kind of harm so quickly from a technology; for some of these users, who it is really having terrible effects on their lives this year.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: Yeah. We\u2019re talking on December 1 here, and yesterday was the third anniversary of the rollout of ChatGPT. Which was, you know, canonically this quote-unquote low-key research preview, right?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Like, it wasn\u2019t supposed to be the phenomenon that it was. It was supposed to be some way for them to, you know, get some users to interact with their large language model and see if they liked it. Again, it was sort of like a trick to see what kind of engagement strategies would work as an interface for large language models.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And when I wrote a piece about this, sort of reflecting on the past three years\u2014and to your point, that what stood out to me is the speed, right? So much has happened in terms of rewiring our economy, our culture. All kinds of different institutions grappling with What do we do now? Technology as universities are a great example, right?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">That this technology has sort of made it so easy to change what it is that we do and to game it. It\u2019s all felt like a speedrunning, and I\u2019m glad you brought up the social-media scenario. It feels like a speedrunning of that. And I\u2019m curious\u2014as somebody, you\u2019ve been reporting on all these big tech companies for such a long time.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">A detail that really stuck out to me\u2014in the bigger story that you just wrote about this\u2014was just a very small detail about Nick Turley. Who was hired, he\u2019s 30 years old, was hired to become the head of ChatGPT. Joined in 2022, and isn\u2019t an AI guy. This is the detail that I thought was really interesting.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">You know, did product stuff at Dropbox and Instacart. I\u2019m wondering\u2014do you feel like you are watching these companies, these tech companies, make the mistakes of 10, 15 years ago? Like almost just over again?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Well, I think one thing I learned in reporting out this story, and talking to a lot of people at OpenAI who have been at OpenAI, is that, you know, the DNA of the place has changed so much.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">A lot of people don\u2019t know, but it was founded in 2015. It was an AI-research lab. It was all about building technology that would benefit humanity. So it was a lot of kind of, like, AI wonks and philosophers and like machine-learning experts who are working on, like, Let\u2019s do something really cool, generative AI.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And then, a lot of people who just like wrote memos and thought about how AI could harm us. Like, that was the kind of DNA of the company. And then ChatGPT was this moment where everything changed. And I mean, OpenAI has gone from being a nonprofit to now being this extremely capitalized, for-profit company, with a lot of pressure in terms of how much investment it\u2019s taken.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And yeah; it needs to prove it\u2019s got the best AI, and it\u2019s hired all of these people from other technology companies, because it has the fastest-growing product, consumer product, in history. And it needs to serve consumers. And so you have people coming from Meta and Google and yeah\u2014like Nick Turley\u2014Instacart, Dropbox.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And they just want to make a product that people can use. And they\u2019re bringing their metrics over with them. And part of how the social-media companies determine whether they have a good product is whether people use it a lot\u2014whether they use it every day. And OpenAI has been very meticulous about saying, We don\u2019t care about time spent. Because this is the metric that social-media companies use: how many hours you sit there with the app. They say, We don\u2019t care about time spent. We just care if this is a useful product that you come back to. But what\u2019s interesting to me is, like, they are going for daily active use; it\u2019s, internally, the metric that everyone uses. It\u2019s, Does this person come back every day? And I don\u2019t think that they\u2019re training against somebody spending eight hours a day there. And so, how do you train for \u201ccome back every day,\u201d but \u201clike, don\u2019t come back for eight hours every day\u201d? I just think it\u2019s hard to do that.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">But yes\u2014I do think that this is a company that\u2019s adapting these similar metrics. I mean, even OpenAI, it\u2019s been reported, you know, they\u2019re starting to test putting ads into ChatGPT. You know, that\u2019s happening internally. I mean, this is the business model of social media.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">It\u2019s \u201ccome back every day,\u201d right. And \u201clook at ads.\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: Well, it\u2019s funny, too, to watch it happen, especially with AI. Because I think, as you described, OpenAI was like a monastery on the hill, right? Like, doing weird stuff and people spending all their days researching, trying to build something that could usher in a style of super-intelligence.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And then when you look at the company\u2019s evolution over the last, even just like the last couple of months, right? You have like a personless slop app, right? There\u2019s just like a TikTok\u2013style clone feed. You have this idea of testing these ads. You have what are essentially just basic tech-company stuff. And that doesn\u2019t suggest to me the sort of \u201cwe are building God\u201d mentality.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah. I mean, where this is surprising to me is that I think the idea was that this AI would be so useful that people would pay for it. And so I think the question now is, like: How useful is this?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And I think that\u2019s something a lot of journalists are trying to report out and that economists are trying to understand. You know, how necessary is this to people\u2019s work? Is it improving the work that they do? These are kind of open questions. And in the meanwhile, yeah, it\u2019s just a question of, like, can we get\u2026 and it is crazy. I was looking back at a story I wrote in January about a woman who fell in love with ChatGPT. Kind of a canary in the coal mine. And at the time, ChatGPT had 300 million users. And now they have, I mean, last I heard they had 800 million weekly users. It\u2019s just a tremendous amount of growth that is happening.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And so, yeah\u2014I mean, it seems like these product people know what they\u2019re doing. It is certainly, if the the goal is to get more people using this app, then yes: mission accomplished.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: So what are these companies trying to do in response? Or we can just narrow it to OpenAI. What is OpenAI trying to do in response to the reporting that you\u2019ve done, the reporting that others have done with all of this? In terms of trying to decrease that style of engagement?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: So there\u2019s some product features that they\u2019ve rolled out, like there\u2019s a nudge now if you spend many hours in the app that will tell you, Do you wanna take a break?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: I want to talk about that nudge, because you tweeted a photo of that nudge that says, \u201cJust checking in.\u201d And I think you were commenting on this\u2014there\u2019s a bit of a dark-pattern user-behavior thing here, right? Like, that one of the things that says \u201cKeep chatting\u201d is already highlighted and looks very pressable.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And then, \u201cThis was helpful\u201d is the other option\u2014which is the, like: Thanks, I\u2019m gonna take a break here. Did that strike you as a BS feature update?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: So the user interface was, yeah, it was like, \u201cYou\u2019ve been here for a long time. Wanna take a break?\u201d And the main answer was, \u201cKeep chatting,\u201d and it was in a black circle.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And then the other thing was, \u201cThis was helpful.\u201d So it\u2019s like not even clear if you click \u201cThis is helpful,\u201d like, what does that do? Though the ChatGPT\u2019s lead of model behavior tweeted at me and said, Oh, that was only active for three days after we launched, and actually we\u2019ve changed it to this. And now it\u2019s a new kind of pop-up that just says, like, \u201cJust checking in; do you need to take a break?\u201d Or something like that. And then there\u2019s an X, that you can close it out if you want. But yes, it did seem \u2026 I don\u2019t know. I was reading through, there\u2019s this social-media-addiction lawsuit where a lot of documents have come out. And some of them were from TikTok, and some of them were about how they\u2019re kind of one of the platforms that pioneered the \u201ctake a break\u201d nudge.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">These documents said, like, Oh, it hasn\u2019t really worked. Most people just keep watching after they get this nudge. So there is this question of: Do the nudges work? But yes, they put a nudge in place. They put parental controls in place, so that parents can link their accounts to their children and get alerts if they\u2019re talking about suicide or self-harm. Which is something\u2014Adam Raine\u2019s mother was like, How is it that he was able to talk about suicide for more than a month with this thing, and it didn\u2019t alert anybody? It didn\u2019t, you know, tell us or call an authority. It was so devastating for her to see that. And then what\u2019s happened on the engineering side is that they have\u2014I don\u2019t know how to say this, technically\u2014rejiggered, you know, ChatGPT so that it pushes back more on delusional thinking.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">One thing that I discovered in my reporting this year is that when conversations go long, the kind of safety guardrails\u2014it happens with all the AI chatbots\u2014the safety guardrails degrade, and they don\u2019t work as well. And so you can \u2026 like, I wrote about a woman who fell in love with ChatGPT. She could have erotic conversations with it, if the conversation went long enough. Adam Raine could talk about suicide, even though you\u2019re not supposed to be able to be asking it for suicidal methods. So they say that they\u2019ve made it so it\u2019s better at checking in on its guardrails, and not kind of engaging in, you know, unsafe conversations.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">They have \u2026 they talk to mental-health experts, to better recognize when somebody who is using the system is in mental distress. And I asked for an example of that. And their head of safety systems, Johannes Heidecke, said that before, if you told ChatGPT, like, I love talking to you; I can just do it forever; I don\u2019t need to sleep, it would be like, Oh, that\u2019s so cool that you don\u2019t need sleep. And now it\u2019ll recognize that, oh\u2014this might be a person who\u2019s having a manic episode. And so it then might say, You know, sleep is actually really important. You should take a break. Like, go get eight hours of sleep. Um, yeah.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">So they have essentially embedded now in the model a better recognition of when somebody might be having kind of\u2014like, be in a spiral of some kind. And I\u2019ve talked to experts who have tested this. And they say, yes, it\u2019s better at recognizing these kind of moments of distress, but it\u2019s best doing it if it\u2019s all in one prompt. As opposed to if it\u2019s spread out over a longer conversation. In that case, it struggles a little bit more to realize. If you kind of drop it in breadcrumbs, it might not realize that you are in a state of mania or psychosis.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: It makes me think a lot about\u2014I\u2019ve done some reporting around these, you know, radicalized mass shooters, right? Who get in these communities on the internet, and things happen. And the thing that you always see, usually, from these types of people is this kind of disappearing from reality, right? Like, not seeing\u2014like they\u2019re not in public as much, right? Or, the thing that some of these \u2026 I\u2019ve talked to people at platforms who\u2019ve gone back and sort of done the forensics of some of these accounts. Of these people who\u2019ve then gone on to commit these acts of violence.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And, one of the things they notice is the use time, right? Like, they can see in the days leading up: more and more and more use, and less and less and less sleep. You know, like people spending whatever on, you know, Discord. Let\u2019s say, spending 20 hours a day, right? And it\u2019s that idea, of that extended use.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And it makes me think\u2014this isn\u2019t really a question\u2014but it\u2019s just so frustrating that these are the types of things you would want, and imagine, people to just at least be considering, right? Like: just having people on staff who are keenly attuned to the psychological effects of the product?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And it seems so strange to me, talking about all these safeguards that are coming in now, that there\u2019s just nobody thinking of some of these things beforehand. Do you feel the same way?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah. Like, you know, it is not unknown that chatbots can have serious effects on people.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">This has been known for a few years now. It is this technology that is so human-like. There\u2019s actually one psychiatrist who wrote a paper in 2023\u2014it was in the Schizophrenia Bulletin\u2014and it was about how these chatbots were gonna cause delusions. People that are susceptible to delusional thinking. He called it, you know, two years before it started manifesting.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Like, people who work in mental health, particularly mental illnesses. They kind of recognize there\u2019s a cognitive dissonance to talking to this thing\u2014that seems so human, that we perceive to be so intelligent. And I just don\u2019t think it was on the company\u2019s radar. They hired their first full-time psychiatrist in March.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">A lot of these companies have started hiring mental-health professionals. I see a lot of my sources on LinkedIn recently, announcing they\u2019ve been hired by a big tech company to work on AI chatbots. So it seems like they\u2019ve awoken to this.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">But, you know, a lot of these companies, when they were first thinking about kind of the risk of AI, it was very much in these kind of existential, Oh, it\u2019s gonna, you know, take over. It\u2019s going to take all of our jobs. Or People are gonna use it to create bioweapons or to hack things. It was all about the damage that people could do with the chatbot\u2014and not the damage the chatbot could do to the person. I\u2019ve read all these early safety papers, and it\u2019s just not there.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">The only version of it you see is they talk about persuasion, or they talk about overreliance. But they talk about persuasion as, Oh, people are gonna use this to develop propaganda that they use to persuade people. Or They\u2019re going to get reliant on this and forget how to think. But it wasn\u2019t like They\u2019re gonna use this to persuade themselves of something that is not true or They\u2019re going to outsource their sense of reality to this chatbot. And, I don\u2019t know, maybe if they\u2019ve had more mental-health professionals kind of involved, maybe they would\u2019ve clocked this. I\u2019m not sure.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: You know, you\u2019ve done so much reporting on these negative externalities of this.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Something that I see as, I guess, pushback\u2014or I\u2019ve seen some OpenAI employees tweet about this\u2014is the notion that there are also a lot of people using these tools as almost, you know, stand-ins for therapists or mental-health professionals. Or just as, like, confidants.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Right. And that this can have a positive effect. I\u2019m certainly not trying to ask you to advocate that this is good, in any way. But are you seeing the opposite of any of what you\u2019ve reported? Are you seeing versions of people just having really good, positive experiences, mental health\u2013wise with this?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Are there two sides to this? Or is this, as you see it, a phenomenon that\u2019s really kind of weighted in the negative direction?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Well, I\u2019ve talked to\u2014I don\u2019t get a lot of emails from people being like, \u201cThis kept me from committing suicide,\u201d or like, \u201cThis has really changed my life for the better.\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And they may well be out there, and they\u2019re just not in my inbox. ChatGPT is not telling them to email me. I\u2019d love to hear from them, more positivity. It\u2019s been a rough year, mental health\u2013wise for me reporting some of these stories. Pretty devastating. But I\u2019ve talked to therapists and psychologists and psychiatrists about this. And they say these tools can be good as a way to kind of think through issues, or process your feelings. You know, it\u2019s almost like an interactive journal. Kind of like writing things out. People are willing to disclose information about themselves to AI chatbots that they wouldn\u2019t tell another human being, because they\u2019re afraid about being judged. As a privacy reporter, that really concerns me.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">That\u2019s another issue. But in terms of mental health, that can be good for people to talk things out that are really difficult. The chatbots perform empathy really convincingly. It doesn\u2019t tire of you like a human would. It has endless reserves to hear about the thing that is bothering you. Like, this is a place where I hear a lot of people say, I had an argument, and I kind of use ChatGPT to process it. I think a lot of people in the therapy space see benefits that could come from AI chatbots. You know, I think the problem is\u2014they can\u2019t offer help. Like, if you really are in a crisis, they can\u2019t do anything to help you. And it\u2019s not \u2026 you know, they\u2019re trying to train it better to react to that kind of thing. But yeah, I mean, a lot of mental-health professionals said there should be better \u201cwarm handoffs,\u201d where it gets you out of the chat.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And \u201cgo talk to a human being,\u201d whether that\u2019s a friend or a family member or a professional. And yeah, I think that\u2019s like a bigger problem with the technology that we use today\u2014it\u2019s so often designed to get us to keep using it, as opposed to push us toward each other. I think it can\u2014like, therapists, they\u2019ve told me this can be a good thing for some people. It\u2019s just the problem is: If they get sucked into it, if they cut off ties with their real human beings, and if they put too much trust in this thing that is a fancy word calculator.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: I love that notion of\u2014I mean, first of all, that this is an engagement problem, like so many engagement problems, right? This notion of just trying to extract more and more and more from your users. And at the end, like, legitimately poisoning the well, right. It\u2019s sort of the classic big-tech problem. I also love that thought of, like, a soft handoff, right? I think that ultimately, we refer to these products all the time as \u201ctools.\u201d But ultimately, a tool should be something that you know, asks you or necessitates putting it down, right?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Or getting you to the place where you can actually have the real fix. Right? And I think that, in that way, these companies are constantly undercutting their own definition as a tool. I\u2019m curious, though. In talking about all this\u2014you know, you are a parent yourself.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">You are somebody who uses technology. You\u2019ve used AI in some of these investigations you\u2019ve done to help organize. What\u2019s your relationship with this product in your life? And are your defenses super-raised at all times?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Like, I gotta make sure that I don\u2019t fall prey to any of these things. Have you ever caught yourself sort of just feeling like this\u2014having an interaction with a chatbot where you\u2019re like, Oh, wow, that felt like a person to me for a split-second.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Because, personally, I\u2019ve asked it to do something, and it\u2019s done it and been really nice about it. And I\u2019m like, Oh man, I gotta close the laptop before I thank it. Or anything like that. But what\u2019s it like in your life?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah; I mean, what honestly turned me on to covering chatbots this year is that in the fall, I guess a year ago, I did this story about turning over all my decision making to generative-AI tools. Like I had it parent for me, and, you know, choose what I wore, choose what I ate, choose what I cook, take us on a vacation. Like, I outsourced everything. And I was trying all the AI chatbots. And I ended up mostly using ChatGPT, because it was more fun. Like, it was the most personable. Gemini seemed a little businesslike. Copilot was pretty bland. Claude was too scold-y. It told me it wasn\u2019t a good idea to outsource my decision making to a chatbot. And ChatGPT was always, like, good to go\u2014willing to make decisions for me.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Actually the office paint behind me\u2014it chose this paint for my office. Like it made relatively good decisions for me, and it named itself Spark. When one of my kids\u2014we were talking about We should give it a name, since it was gonna be with us all week. And we had it in voice mode, and my daughters were like, \u201cLet\u2019s name it Captain Poopy Head.\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And it was like, \u201cActually, Spark would be a better name for me.\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: Put some respect on it.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah. And so Spark became this kind of entity. Like, kind of a person. And my kids still like to talk to Spark. And we were using it one day, and they were asking Spark, \u201cWhat\u2019s your favorite food?\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And it was like, \u201cMy favorite food is pizza, because I love how gooey and cheesy it is.\u201d And I just had this visceral reaction of horror that it was saying it had a favorite food. Because it should say, \u201cI\u2019m a large language model. I don\u2019t eat food.\u201d And it just is this recognition that\u2014yeah, my kids will like this. They\u2019re in that dissonant phase.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And so we, I talk about this with my kids. Like, Oh, you know, that\u2019s an AI chatbot. Like, it\u2019s not real. It\u2019s really good at finding information. I enjoy it for data analysis. I use it for, \u201cThere\u2019s a problem in my house; how do I fix this?\u201d My kids have a medical issue. It\u2019s actually quite good at giving health information, though it can make horrible mistakes.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">So I wouldn\u2019t use it for anything too serious. I always think of the guy who wanted to cut salt out of his diet, and it had him start eating bromide. And then he had\u2014I mean, that was a direct psychotic breakdown from advice from a chatbot. But you know, it can give you bad advice. So I do think of it as better than Google.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">You know, the search engines are so cluttered now that I find [a chatbot] a better place sometimes to go. But I think of it as a good place to start a search\u2014and not end a search. But yeah; I think, like, it\u2019s a generative-AI tool, that it\u2019s a good tool. It\u2019s good for data analysis. It\u2019s good for getting information off the internet.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">I don\u2019t want it to be my best friend. I don\u2019t want everybody to have AI friends. I want people to have real people in their life that they spend most of their time with.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: Well, this sort of brings me to the place where I want to close. Which is: Some of the work that you\u2019ve done over the years that&#8217;s been really canonical for us tech reporters is a series that you did trying to expose just how locked in we are to all these various platforms, right? And how difficult it is to try to leave them and get them out of your lives in whatever fashion\u2014be it Google, Amazon, whatever. And something that I think about with these chatbots that you bring up, right, is: People are using them as replacements for some of these really big tech tools like Google, right? Like, people are using them as their search engines. You know, these AI companies want to eat the browser. They want to, you know, get you inside this walled-garden experience and have it do all of the things, right? This is sort of what we\u2019re talking about, and I wonder how you think about the future. Given that there is this usefulness\u2014there is this desire on the behalf of these companies to want to lock you in, to want to keep you engaged. And there is, as we can see, this emergent behavior that humans have, when they get stuck in these spirals with these types of chatbots, potentially. And so it seems to me like we\u2019re just creating a situation where there\u2019s gonna be more and more users, and more and more lock-in, and more and more pressure put on everyday people to interact in these specific ways that could lead to these problematic, you know, delusional sort of outcomes. Do you see this? This problem that we\u2019re shorthanding as \u201cAI psychosis,\u201d even though we\u2019re not calling it psychosis. Do you see that?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Like, are you worried this is gonna be a bigger and bigger and bigger problem going forward because of where the technology is headed? And where we are sort of socially headed, using it?<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Yeah; I just think it\u2019s gonna be an acceleration of existing trends we\u2019ve seen in technology, where you will get this very personalized version of the world.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">You know\u2014some people have described these things as improv actors, like they are reacting. Every version of ChatGPT is different. Probably if you ask if it has a favorite food, it might not\u2014or it has a different favorite food. Like it is personalized to you. And so I\u2019m imagining this world in which everybody has this agentic AI., They have this, like, version of the world that\u2019s fed through an AI chatbot that is personalized to them.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">And that I have two fears. One is: It flattens us, flattens us all out. It makes us very boring, because we all are getting like a version of the same advice. That\u2019s kind of what I came away from when I lived on it for a week. I was like, \u201cWow, I feel really boring\u201d at the end. \u201cI feel like the most basic version of myself.\u201d<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">The other version is that it makes each one of us eccentric in a new way, where it gets so personalized to us that it moves us farther and farther away from other people. The way I\u2019ve been thinking about kind of the delusion stuff is the way that some celebrities or billionaires have these sycophants around them who tell them that every idea they have is<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">brilliant. And, you know, they\u2019re just surrounded by yes-men. What AI chatbots are is like your personal sycophant, your personal yes-man, that will tell you your every idea is brilliant. And in the same way that these celebrities and billionaires can become quite eccentric and quite antisocial\u2014some of them, I think, some people are more susceptible to than others\u2014this could happen to all of us. Right. Those are the two things I\u2019m really afraid of. Either it makes us all incredibly bland, where we\u2019re all speaking ChatGPT-ese and, like, you can\u2019t tell the difference between anybody who\u2019s in your inbox. Or we all kind of move in this very polarized, eccentric direction, where we can\u2019t be as kind to one another, as human beings. So yeah: Those are the two dystopias. Or maybe I\u2019m totally wrong, and it\u2019ll just make the world wonderful for all of us.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: \u201cThe billionaires have democratized the experience of being a billionaire,\u201d I think, is a wonderful place to leave it. It&#8217;s perfectly dystopian for the Galaxy Brain podcast here. Kashmir Hill, thank you so much for your time, your reporting, and your insights on all this.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Hill: Thanks so much for inviting me on.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">Warzel: That\u2019s it for us here. Thank you again to my guest, Kashmir Hill. If you liked what you saw here, new episodes of Galaxy Brain drop every Friday. You could subscribe to The Atlantic\u2019s YouTube channel, or on Apple or Spotify or wherever you get your podcasts. And if you enjoyed this, remember, you can support the work of myself and other journalists at The Atlantic by subscribing to the publication at TheAtlantic.com\/Listener.<\/p>\n<p class=\"ArticleParagraph_root__4mszW\" data-flatplan-paragraph=\"true\">That\u2019s TheAtlantic.com\/Listener. Thanks again, and see you on the internet.<\/p>\n","protected":false},"excerpt":{"rendered":"Subscribe here: Apple Podcasts | Spotify | YouTube In this episode of Galaxy Brain, Charlie Warzel explores the&hellip;\n","protected":false},"author":2,"featured_media":331290,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[62,276,277,49,48,61],"class_list":{"0":"post-331289","1":"post","2":"type-post","3":"status-publish","4":"format-standard","5":"has-post-thumbnail","7":"category-artificial-intelligence","8":"tag-ai","9":"tag-artificial-intelligence","10":"tag-artificialintelligence","11":"tag-ca","12":"tag-canada","13":"tag-technology"},"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/331289","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=331289"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/331289\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/331290"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=331289"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=331289"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=331289"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}