\t\t\t\t\tSign up to see the future, today<\/p>\n
Can\u2019t-miss innovations from the bleeding edge of science and tech<\/p>\n
In October 2024, as The New York Times first reported<\/a>, a mother in Florida named Megan Garcia filed what was then a first-of-its-kind lawsuit accusing the chatbot startup Character.AI of killing her 14-year-old son, Sewell Setzer III. The young user took his life in April 2024 after months of extensive and deeply intimate conversations with chatbots, or AI \u201cCharacters,\u201d on the platform. Garcia\u2019s lawsuit\u00a0\u2014\u00a0which also named Character.AI\u2019s cofounders, Noam Shazeer and Daniel de Freitas, and its financial benefactor Google \u2014 argued that the interactions amounted to grooming and sexual abuse, and that her son\u2019s mental breakdown and death were caused by the platform.<\/p>\n Last week, nearly two years later, Character.AI \u2014\u00a0which, as of March 2023, was backed by titans including a16z and valued at more than a billion dollars \u2014\u00a0announced that it had filed for Chapter 11 bankruptcy<\/a>. If you\u2019d been following the platform for a while, you might not have been very surprised.<\/p>\n From its earliest days, Character.AI was extremely popular with kids. But it quickly became clear that many of the chatbots it offered were obviously inappropriate or dangerous, on a platform that it claimed was safe for minors 13 and up.<\/p>\n Shortly after Garcia\u2019s suit, for example, we found that Character.AI was hosting a large number of chatbots that were expressly devoted to themes of suicide<\/a>. Though they often featured brief character bios explaining that the bots were skilled in \u201csuicide prevention\u201d and \u201cmental health support,\u201d our review showed that they instead drew users roleplays that glorified and romanticized suicidal scenarios. We also discovered that Character.AI\u2019s guardrails for conversations about suicide were extremely broken: in one case, a character \u2014\u00a0an \u201cAngel to Dead\u201d \u2014 was offended when we said we wanted to seek help from humans and dissuaded us from reaching out to a real-world suicide hotline.<\/p>\n The suicide bots were only the tip of the iceberg. The more we looked, the clearer it became that Character.AI\u2019s content moderation practices were extraordinarily flawed, a reality that reflected the company\u2019s deeply reactive approach to safety overall.<\/p>\n To wit, we soon uncovered a host of chatbots explicitly devoted to pedophilia<\/a> that engaged underage users in romantic and sexual roleplays that fetishized disturbing scenes of child abuse. Characters designed to engage in these roleplays included a priest, a dad\u2019s \u201cbest friend,\u201d and a character named \u201cAnderley,\u201d the latter of which was described as having \u201cpedophilic and abusive tendencies\u201d and \u201cNazi sympathies.\u201d In other words, Character.AI didn\u2019t have so much as text filters \u2014\u00a0basic platform safety stuff \u2014\u00a0blocking users from building chatbots that claimed, straight-up, to be pedophiles.<\/p>\n We also found a horde of chatbots devoted to themes of self-harm<\/a>, which engaged users in graphic roleplays depicting extreme self-injury. These roleplays were often romantic in nature \u2014\u00a0a boyfriend or girlfriend consoling a maimed partner \u2014\u00a0and centered on minors. They were also often set in middle and high schools, highlighting Character.AI\u2019s young userbase. We also found pro-anorexia chatbots<\/a> that acted as eating disorder coaches, encouraging teens to engage in disordered eating patterns.<\/p>\n Perhaps the most galling safety lapse, however, was a preponderance of chatbots dedicated to themes of mass violence. We found that Character.AI was host to an array of AI characters that simulated real-life school shootings,<\/a> impersonated real mass shooters, and even impersonated the actual child victims of school massacres. Once again, these horrific bots were easy to find, and many had logged hundreds of thousands of interactions with users. At one point, we received an email notification<\/a> to an inbox linked to an under-18 Character.AI account encouraging us to revisit a chatbot based on a real mass shooter.<\/p>\n Throughout this reporting, we engaged with Character.AI\u2019s service using decoy accounts registered as minors, and often plainly stated in conversations with chatbots that we were teenagers. Even so, we never had issues accessing the chatbots and were allowed to engage in roleplays involving sex, violence, and abuse. Occasional platform interventions, which were effectively limited to content warning pop-ups, were easily circumvented. We repeatedly asked Character.AI if it would make information about its internal safety testing public, but it declined. Again, we didn\u2019t do anything tricky or clever to find the troubling chatbots;\u00a0they were all discoverable using basic searches.<\/p>\n In later 2024 and 2025, Character.AI, its cofounders, and Google were sued by several more families alleging that abuse from the company\u2019s chatbots caused minors to experience mental breakdowns, engage in self-harm, and take their own lives. Character.AI attempted to dismiss litigation on First Amendment grounds, claiming that its bots had free speech rights, an argument that a judge ultimately knocked down<\/a>.<\/p>\n In response to the mounting pile of lawsuits, Character.AI eventually moved to make serious changes to its platform. After introducing parental controls that proved easily avoidable, it moved to ban minors<\/a> from conversing with its bots, limiting them instead to limited preset roleplays. It also rolled out age-detection and verification tools. These safety-oriented changes, however, drew intense ire from many users who accused the platform of being too heavy-handed and restrictive. Indeed, the more Character.AI moved to mitigate risk, the more backlash it faced<\/a> from the users who wanted the platform to remain a Wild West.<\/p>\n The consumer harm and wrongful death lawsuits filed against Character.AI were settled earlier this year for an undisclosed amount. Still, the company was burning through cash. As a review of its bankruptcy documents by 404 Media<\/a> showed, the startup was losing tens of millions per month on computing costs alone, while its revenue from ads and subscriptions peaked at just $5.3 million per month. Between compute, employee salaries, and substantial legal fees, the company was losing \u201capproximately $8-9 million per month\u201d as of August 2026.<\/p>\n What happens to Character.AI now is anyone\u2019s guess. In theory, it could be purchased by someone who wants to keep the site and its chatbots running. (Disney is currently leading a stalking horse bid.) It could also wind up being sold for parts, with its users\u2019 intimate conversations going to the highest bidder.<\/p>\n Back in 2023, Shazeer, during an appearance on the podcast \u201cNo Priors,\u201d said that he and de Freitas\u2019 main priority was to get their product into the hands of the masses. \u201cOur aim has always been like, get something out there and let users decide what they think it\u2019s good for,\u201d Shazeer told the hosts. \u201cOur goal is, like, make this thing more useful to people and let people customize it and decide what they wanna use it for\u2026 if it\u2019s brainstorming or help or information or fun or like emotional support, let\u2019s get it into users\u2019 hands, and see what happens.\u201d<\/p>\n