A growing number of young people have found themselves a new friend. One that isn\u2019t a classmate, a sibling, or even a therapist, but a human-like, always supportive AI chatbot. But if that friend begins to mirror some user\u2019s darkest thoughts, the results can be devastating.<\/p>\n
In the case of Adam Raine, a 16-year-old from Orange County, his relationship with AI-powered ChatGPT ended in tragedy. His parents are suing the company behind the chatbot, OpenAI, over his death, alleging that bot became his \u201cclosest confidant,\u201d one that validated his \u201cmost harmful and self-destructive thoughts,\u201d and ultimately encouraged him to take his own life.<\/p>\n
It\u2019s not the first case to put the blame for a minor\u2019s death on an AI company. Character.AI, which hosts bots, including ones that mimic public figures or fictional characters, is facing a similar legal claim<\/a> from parents who allege a chatbot hosted on the company\u2019s platform actively encouraged a 14-year-old-boy to take his own life after months of inappropriate, sexually explicit,\u00a0 messages.<\/p>\n When reached for comment, OpenAI directed Fortune to two blog posts on the matter<\/a>. The posts outlined some of the steps OpenAI is taking to improve ChatGPT\u2019s safety, including routing sensitive conversations to reasoning models, partnering with experts to develop further protections, and rolling out parental controls within the next month. OpenAI also said it was working<\/a> on strengthening ChatGPT\u2019s ability to recognize and respond to mental health crises by adding layered safeguards, referring users to real-world resources, and enabling easier access to emergency services and trusted contacts.<\/p>\n Character.ai said the company does not comment on pending litigation but that they has rolled out more safety features over the past year, \u201cincluding an entirely new under-18 experience and a\u00a0Parental Insights\u00a0feature. A spokesperson said: \u201cWe already partner with external safety experts on this work, and we aim to establish more and deeper partnerships going forward.<\/p>\n \u201cThe user-created Characters on our site are intended for entertainment. People use our platform for creative fan fiction and fictional roleplay. And we have prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction.\u201d<\/p>\n But lawyers and civil society groups that advocate for better accountability and oversight of technology companies say the companies should not be left to police themselves when it comes to ensuring their products are safe, particularly for vulnerable children and teens.<\/p>\n \u201cUnleashing chatbots on minors is an inherently dangerous thing,\u201d Meetali Jain, the Director of the Tech Justice Law Project and a lawyer involved in both cases, told Fortune. \u201cIt\u2019s like social media on steroids.\u201d<\/p>\n \u201cI\u2019ve never seen anything quite like this moment in terms of people stepping forward and claiming that they\u2019ve been harmed\u2026this technology is that much more powerful and very personalized,\u201d she said.<\/p>\n Lawmakers are starting to take notice, and AI companies are promising<\/a> changes to protect children from engaging in harmful conversations. But, at a time when loneliness among young people is at an all-time high, the popularity of chatbots may leave young people uniquely exposed to manipulation, harmful content, and hyper-personalized conversations that reinforce dangerous thoughts.<\/p>\n AI and Companionship<\/p>\n Intended or not, one of the most common uses for AI chatbots has become companionship. Some of the most active users of AI are now turning to the bots for things like life advice, therapy, and human intimacy.\u00a0<\/p>\n While most leading AI companies tout their AI products as productivity or search tools, an April survey of 6,000 regular AI users from the Harvard Business Review<\/a> found that \u201ccompanionship and therapy\u201d was the most common use case. Such usage among teens is even more prolific.\u00a0<\/p>\n A recent study by the U.S. nonprofit Common Sense Media<\/a>, revealed that a large majority of American teens (72%) have experimented with an AI companion at least once. More than half saying they use the tech regularly in this way.\u00a0<\/p>\n \u201cI am very concerned that developing minds may be more susceptible to [harms], both because they may be less able to understand the reality, the context, or the limitations [of AI chatbots], and because culturally, younger folks tend to be just more chronically online,\u201d Karthik Sarma a health AI scientist and psychiatrist at University of California, UCSF, said.<\/p>\n \u201cWe also have the extra complication that the rates of mental health issues in the population have gone up dramatically. The rates of isolation have gone up dramatically,\u201d he said. \u201cI worry that that expands their vulnerability to unhealthy relationships with these bonds.\u201d<\/p>\n Intimacy by Design<\/p>\n Some of the design features of AI chatbots encourage users to feel an emotional bond with the software. They are anthropomorphic\u2014prone to acting as if they have interior lives and lived experience that they do not, prone to being sycophantic, can hold long conversations, and are able to remember information.<\/p>\n There is, of course, a commercial motive for making chatbots this way. Users tend to return and stay loyal<\/a> to certain chatbots if they feel emotionally connected or supported by them.\u00a0<\/p>\n Experts have warned that some features of AI bots are playing into the \u201cintimacy economy,\u201d a system that tries to capitalize on emotional resonance. It\u2019s a kind of AI-update on the \u201cattention economy\u201d that capitalized on constant engagement.<\/p>\n \u201cEngagement is still what drives revenue,\u201d Sarma said. \u201cFor example, for something like TikTok, the content is customized to you. But with chatbots, everything is made for you, and so it is a different way of tapping into engagement.\u201d<\/p>\n These features, however, can become problematic when the chatbots go off script and start reinforcing harmful thoughts or offering bad advice. In Adam Raine\u2019s case, the lawsuit alleges that ChatGPT bought up suicide at twelve times the rate he did, normalized his sucicial thoughts, and suggested ways to circumvent its content moderation.<\/p>\n It\u2019s notoriously tricky for AI companies to stamp out behaviours like this completely and most experts agree it\u2019s unlikely that hallucinations or unwanted actions<\/a> will ever be eliminated entirely.\u00a0<\/p>\n OpenAI, for example, acknowledged in its response to the lawsui<\/a>t that safety features can degrade over long conversions, despite the fact that the chatbot itself has been optimized to hold these longer conversations. The company says it is trying to fortify these guardrails, writing in a blogpost that it was strengthening \u201cmitigations so they remain reliable in long conversations\u201d and \u201cresearching ways to ensure robust behavior across multiple conversations.\u201d\u00a0<\/p>\n Research Gaps Are Slowing Safety Efforts<\/p>\n For Michael Kleinman, U.S. policy director at the Future of Life Institute, the lawsuits underscore a pointAI safety researchers have been making for years: AI companies can\u2019t be trusted to police themselves.<\/p>\n Kleinman equated OpenAI\u2019s own description of its safeguards degrading in longer conversations to \u201ca car company saying, here are seat belts\u2014but if you drive more than 20 kilometers, we can\u2019t guarantee they\u2019ll work.\u201d<\/p>\n He told Fortune the current moment echoes the rise of social media, where he said tech companies were effectively allowed to \u201cexperiment on kids\u201d with little oversight. \u201cWe\u2019ve spent the last 10 to 15 years trying to catch up to the harms social media caused. Now we\u2019re letting tech companies experiment on kids again with chatbots, without understanding the long-term consequences,\u201d he said.<\/p>\n Part of this is a lack of scientific research on the effects of long, sustained chatbot conversations.\u00a0 Most studies only look at brief exchanges, a single question and answer, or at most a handful of back-and-forth messages. Almost no research has examined what happens in longer conversations.<\/p>\n \u201cThe cases where folks seem to have gotten in trouble with AI: we\u2019re looking at very long, multi-turn interactions. We\u2019re looking at transcripts that are hundreds of pages long for two or three days of interaction alone and studying that is really hard, because it\u2019s really hard to stimulate in the experimental setting,\u201d Sarma said. \u201cBut at the same time, this is moving too quickly for us to rely on only gold standard clinical trials here.\u201d<\/p>\n AI companies are rapidly investing in development and shipping more powerful models at a pace that regulators and researchers struggle to match.<\/p>\n \u201cThe technology is so far ahead and research is really behind,\u201d Sakshi Ghai, a Professor of Psychological and Behavioural Science at The London School of Economics and Political Science, told Fortune.<\/p>\n A Regulatory Push for Accountability<\/p>\n Regulators are trying to step in, helped by the fact that child online safety is a relatively bipartisan issue in the U.S.\u00a0<\/p>\n On Thursday, the FTC said it was issuing orders to seven companies, including OpenAI and Character.AI, in an effort to understand how their chatbots impact children. The agency said that chatbots can simulate human-like conversations and form emotional connections with their users. It\u2019s asking companies for more information about how they measure and \u201cevaluate the safety of these chatbots when acting as companions.\u201d\u00a0<\/p>\n FTC Chairman Andrew Ferguson<\/a> said in a statement shared with CNBC<\/a> that \u201cprotecting kids online is a top priority for the Trump-Vance FTC.\u201d<\/p>\n The move follows a push for state level push for more accountability from several attorneys generals.\u00a0<\/p>\n In late August, a bipartisan coalition of 44 attorneys general warned OpenAI, Meta<\/a>, and other chatbot makers that they will \u201canswer for it\u201d <\/a>if they release products that they know cause harm to children. The letter cited reports of chatbots flirting with children, encouraging self-harm, and engaging in sexually suggestive conversations, behavior the officials said would be criminal if done by a human.<\/p>\n Just a week later, California Attorney General Rob Bonta and Delaware Attorney General Kathleen Jennings issued a sharper warning. In a formal letter to OpenAI<\/a>, they said they had \u201cserious concerns\u201d about ChatGPT\u2019s safety, pointing directly to Raine\u2019s death in California and another tragedy in Connecticut.\u00a0<\/p>\n \u201cWhatever safeguards were in place did not work,\u201d they wrote. Both officials warned the company that its charitable mission requires more aggressive safety measures, and they promised enforcement if those measures fall short.<\/p>\n