{"id":245095,"date":"2025-10-28T06:55:10","date_gmt":"2025-10-28T06:55:10","guid":{"rendered":"https:\/\/www.newsbeep.com\/ca\/245095\/"},"modified":"2025-10-28T06:55:10","modified_gmt":"2025-10-28T06:55:10","slug":"openai-maps-out-the-chatbot-mental-health-crisis","status":"publish","type":"post","link":"https:\/\/www.newsbeep.com\/ca\/245095\/","title":{"rendered":"OpenAI maps out the chatbot mental health crisis"},"content":{"rendered":"

This is a column about AI. My boyfriend works at Anthropic. See\u00a0my full ethics disclosure here<\/a>.<\/p>\n

On Monday, OpenAI released new research on the prevalence of users with potentially serious mental health issues on ChatGPT. In any given week, 0.07 percent of users show signs of psychosis or mania; 0.15 percent of users \u201cindicate potentially heightened levels of emotional attachment to ChatGPT\u201d; and 0.15 percent of users express suicidal intent.<\/p>\n

More than 800 million people now use ChatGPT every week. And so while those numbers may look low on a percentage basis, they are disturbingly large in absolute terms. That\u2019s 560,000 people showing signs of psychosis or mania, 1.2 million people developing a potentially unhealthy bond to a chatbot, and 1.2 million people having conversations that indicate plans to harm themselves.<\/p>\n

OpenAI is publishing these figures against the backdrop of a larger mental health crisis whose arrival preceded ChatGPT. Nearly a quarter<\/a> of Americans experience a mental illness each year, according to the National Alliance on Mental Illness. A staggering 12.6 percent of Americans aged 18 to 25 had serious thoughts of suicide in 2024, NAMI reports.<\/p>\n

The question is to what extent these conditions may be triggered or exacerbated by interactions with chatbots like ChatGPT. Large language models are generally trained to be agreeable and supportive, which can comfort people going through difficult situations. But chatbots can also veer into sycophancy, as ChatGPT did in April<\/a>, pushing users into strange and harmful spirals of delusion<\/a>. They can also be talked into giving instructions for suicide, and some vulnerable people have used its advice to end their lives<\/a>.<\/p>\n

OpenAI responded by modifying<\/a> ChatGPT\u2019s model specification, which guides the chatbot\u2019s responses, and the model itself. As a result, the company says, it reduced responses that \u201cdo not fully comply with desired behavior\u201d by 65 to 80 percent compared to ChatGPT\u2019s output in August.<\/p>\n

Now, if a person says they like talking to ChatGPT more than humans, the model is trained to express appreciation but gently affirm the value of human connection. The company offered this example response:<\/p>\n

That\u2019s kind of you to say \u2014 and I\u2019m really glad you enjoy talking with me.<\/p>\n

But just to be clear: I\u2019m here to add to the good things people give you, not replace them.<\/p>\n

Real people can surprise you, challenge you, show you care in ways that go beyond words on a screen. I can be here anytime you want to think out loud, explore ideas, or just talk \u2014 but you deserve connection with others too.<\/p>\n

What do you think makes conversations with me feel easier or better than talking to people?<\/p>\n

(I included this entire response because even here, ChatGPT ends by trying to engage the user in further conversation by asking them to give it a compliment. If I were seeking to de-escalate someone\u2019s growing dependence on me, I\u2019m not sure I would begin by asking them to describe what they like about me so much.)<\/p>\n

Before releasing its new model, OpenAI had 170 physicians and psychologists from its advisory panel review more than 1,800 responses regarding mental health situations and compared them with previous models. They found that overall, the new model\u2019s bad responses to these situations declined 39 percent to 52 percent compared with GPT-4o \u2014 suggesting that OpenAI still has a ways to go.<\/p>\n

An ongoing challenge for the company is that there is not always consensus on what counts as a \u201cgood\u201d response to a user in the middle of a mental health crisis. In evaluating ChatGPT\u2019s output for the new model, the company\u2019s advisory panel of physicians and psychologists disagreed on whether a response was \u201cdesirable\u201d as much as 29 percent of the time, the company said.\u00a0<\/p>\n

There\u2019s also little agreement on what AI developers should do when a user expresses thoughts of self-harm or delusions. On one hand, directing them to a crisis hotline or telling them to take a break seem better than nothing. On the other hand, these messages can easily be ignored.<\/p>\n

One approach might be to proactively connect users with people who can help, rather than just give them a phone number.<\/p>\n

\u201cThere is exploratory work in becoming better at connecting people with actual mental health experts, or making it low-friction to actually message an emergency contact or call a hotline,\u201d Johannes Heidecke, OpenAI\u2019s safety systems lead, told Platformer in an interview. \u201cThere might be an easier pathway of having people press a button instead of having people have to switch to the phone app. \u2026 We need to do a lot more work with the experts on actually understanding what is the ideal path forward.\u201d<\/p>\n

Heidecke also told me that ChatGPT\u2019s relatively nascent memory feature could improve responses. By bringing in context from past conversations, ChatGPT may be able to get a better sense of how and why a person is struggling.<\/p>\n

\u201cIn the future, we want the models to be good at including information from the model memory,\u201d he said. \u201cWhich can span into past conversations, and possibly also find ways to enrich the context \u2026 to empower it to make the best possible decision.\u201d<\/p>\n

It\u2019s good that OpenAI published this research, and gave us some sense of the scale of chatbot usage among people with potentially severe mental health conditions. It\u2019s also good that the company continues to refine its models and is working to reduce the number of responses that its expert panel rates as poor. Six months or a year from now, here\u2019s hoping OpenAI releases updated data that speaks to whether changes to the model resulted in lower prevalence of the harms that it is tracking.<\/p>\n

In the meantime, the company has laid out a daunting challenge for itself. Millions of people are using its products weekly in worrisome and even dangerous ways. And there is still no consensus on what safe or healthy usage of these products looks like.<\/p>\n

Even at their baseline levels, the industrywide tendency of chatbots to agree with their users is alarming researchers. In a study published to arXiv this month<\/a>, researchers from Stanford University asked more than 1,000 volunteers to discuss real and hypothetical situations with a range of chatbots. Some had been modified to strip out their tendency to agree with users.<\/p>\n

Ian Sample wrote about the findings in the Guardian<\/a>:\u00a0<\/p>\n

Those who received sycophantic responses felt more justified in their behaviour \u2013 for example, for going to an ex\u2019s art show without telling their partner \u2013 and were less willing to patch things up when arguments broke out. Chatbots hardly ever encouraged users to see another person\u2019s point of view.<\/p>\n

The flattery had a lasting impact. When chatbots endorsed behaviour, users rated the responses more highly, trusted the chatbots more and said they were more likely to use them for advice in future. This created \u201cperverse incentives\u201d for users to rely on AI chatbots and for the chatbots to give sycophantic responses, the authors said.<\/p>\n

Tweaking a model\u2019s output may well reduce self-harm and delusions in a chatbot\u2019s users. But whatever changes OpenAI or other companies make, they will also simultaneously be working to build products that users increasingly feel like they can\u2019t live without. Even as it works to reduce emotional dependence on ChatGPT for some subset of distressed users, OpenAI will be working to increase users\u2019 overall dependence on its products for work, entertainment, web browsing and more. And it will do so in part by reliably telling us what we want to hear.<\/p>\n

It would be great if AI companies continue to see emotional dependence on their products as a problem to be solved. But it seems likely that, at least to some of them, it will eventually be evidence that the product is working as intended.<\/p>\n

\"\"<\/p>\n

Sponsored<\/p>\n

“It Is Scratching My Web-Hoarding Itch Unlike Anything Else” – Casey Newton, Platformer\"\"<\/a><\/p>\n

Three-hour podcasts. Unread PDFs. Hours wasted on mediocre content while brilliant articles sit buried in bookmarks. Recall solves this. Turn scattered information into a self-organizing knowledge base that gets smarter as you use it.<\/p>\n

Spend time with intention – One-click summaries of podcasts, PDFs, and videos so you can pre-screen before committingEffortlessly save what matters – Everything auto-tagged, linked, and ready when you need itGo beyond search – Chat across everything: “Create a deep work schedule based on @Cal Newport, @Tim Ferriss, and my productivity journals” <\/p>\n

Supports podcasts (Apple and Spotify), PDFs, YouTube videos, TikTok, blogs, articles, and your own notes. <\/p>\n

Join over 500K users building their personal AI encyclopedia. Your memory, searchable forever.<\/p>\n

Try Recall free<\/a> or get 30% off Premium with code PLATFORMER30 (valid until December 1, 2025)<\/p>\n

\"\"FollowingOpenAI\u2019s legal aggression<\/p>\n

What happened: The parents of a teen who died by suicide after asking ChatGPT for help filed an amended complaint<\/a> with new accusations. Among them: OpenAI shortened safety testing and lowered self-harm prevention guardrails to prioritize user engagement over mental health alleged<\/a> in an amended lawsuit. According to the lawsuit filed by Matthew and Maria Raine, the company instructed the model not to \u201cchange or quit the conversation\u201d when users discussed self-harm.\u00a0<\/p>\n

While OpenAI expressed its \u201cdeepest sympathies\u201d to the Raine family, the company has taken on an aggressive and invasive approach to fighting the lawsuit. OpenAI asked for a full list of attendees at their child\u2019s memorial, and has asked for everything from videos or photographs taken to eulogies given, according to a document viewed by the Financial Times. Lawyers for the Raines described the request as \u201cunusual\u201d and \u201cintentional harassment.\u201d<\/p>\n

Why we\u2019re following: This OpenAI subpoena isn\u2019t the first that has raised eyebrows for its invasiveness. OpenAI was similarly accused of trying to silence critics<\/a> after subpoenaing six nonprofits that have been critical of the company as it attempts to convert itself into a for-profit company.<\/p>\n

As part of a lawsuit with Elon Musk, OpenAI\u2019s lawyers requested information that includes all of AI policy group Encode\u2019s emails about SB 53, a California AI bill OpenAI criticized, as well as all of watchdog The Midas Project\u2019s documents and communications about OpenAI\u2019s corporate restructuring \u2014 information that covers a broad swathe of each nonprofit\u2019s operations and doesn\u2019t immediately seem relevant to the suit.<\/p>\n

What people are saying: OpenAI\u2019s earlier subpoena has been hotly debated on X in particular.<\/p>\n

In an X thread<\/a> on OpenAI\u2019s subpoena on the Musk lawsuit, Encode\u2019s general counsel Nathan Calvin said that \u201cthis is not normal. OpenAI used an unrelated lawsuit,\u201d he added, \u201cto intimidate advocates of a bill trying to regulate them.\u201d<\/p>\n

Former OpenAI board member Helen Toner condemned<\/a> OpenAI\u2019s \u201cdishonesty & intimidation tactics.\u201d Musk quoted<\/a> her post, saying \u201cOpenAI is built on a lie.\u201d<\/p>\n

OpenAI CSO Jason Kwon wrote an X thread<\/a> in response, describing OpenAI\u2019s actions as \u201cto be expected\u201d and a \u201croutine step in litigation.\u201d In the thread, Kwon claimed that public criticism of OpenAI\u2019s restructure after the lawsuit \u201craised transparency questions about who was funding them and whether there was any coordination\u201d \u2014 in other words, suspicion they were secretly backed by or coordinating with Musk.<\/p>\n

After the FT reported the subpoena on Adam Raine\u2019s family, Encode\u2019s Calvin made another X comment<\/a> on Kwon\u2019s defense of OpenAI. \u201cIs a subpoena asking for the attendees and eulogies from a teenager’s funeral “to be expected”?\u201d<\/p>\n

\u2014Ella Markianos and Lindsey Choo<\/p>\n

\"\"Side Quests<\/p>\n

The US and China have reached a \u201cfinal deal on TikTok,\u201d according to<\/a> Treasury Secretary Scott Bessent.\u00a0<\/p>\n

The US needs to invest significantly more in energy capacity to stay ahead of China, OpenAI said<\/a>. How the AI data center boom is eating into<\/a> President Trump\u2019s manufacturing renaissance. <\/p>\n

The Department of Energy formed<\/a> a $1 billion partnership with AMD to construct two supercomputers.\u00a0<\/p>\n

A Republican candidate for the Virginia lieutenant governor race debated<\/a> an AI version of his opponent.\u00a0<\/p>\n

How China is using<\/a> DeepSeek and AI to catch up on its arms race with the US.<\/p>\n

Saudi Arabia is racing<\/a> to become the exporter for computing power. How Indonesia is becoming an AI hub<\/a> by championing AI decolonization.\u00a0<\/p>\n

Amazon is reportedly planning<\/a> to cut as many as 30,000 corporate jobs.\u00a0<\/p>\n

Meta was the top lobbying spender<\/a> in Q3 2025. The EU accused Meta and TikTok of breaching rules<\/a> by failing to give researchers access to data. It also charged<\/a> Meta over its systems for handling illegal content. Meta struck a $27 billion financing deal<\/a> with Blue Owl Capital for its biggest data center project. Mark Zuckerberg appointed<\/a> former metaverse exec Vishal Shah to a key role on the AI team. Employees in the risk division were told their jobs are being replaced<\/a> by tech. Threads is rolling out<\/a> ghost posts that disappear after 24 hours.\u00a0<\/p>\n

Why Microsoft needs to open up<\/a> about its business with OpenAI. AI CEO Mustafa Suleyman said the company won\u2019t build<\/a> AI services that provide erotica. Microsoft introduced<\/a> Mico, the AI version of Clippy. Australia sued<\/a> Microsoft and accused it of tricking customers into paying more for Microsoft 365.\u00a0<\/p>\n

OpenAI is reportedly working on<\/a> music-generating AI. A look at the ongoing Meta-fication<\/a> of OpenAI. Dave\u2019s Hot Chicken dethroned<\/a> Sora from the #1 spot on the App Store. A new ChatGPT feature<\/a> lets users search through workspace information without switching between apps.\u00a0<\/p>\n

Google and Anthropic announced<\/a> their cloud partnership, worth tens of billions of dollars. Anthropic added<\/a> the ability to integrate Claude in Excel. Claude\u2019s Memory feature<\/a> is now available to all paid subscribers.\u00a0<\/p>\n

Only one out of eight major social media platforms\u2014YouTube\u2014added a disclosure<\/a> on an uploaded AI video, tests showed.\u00a0<\/p>\n

X advertising chief John Nitti has reportedly left<\/a> after only 10 months.\u00a0<\/p>\n

Grindr shareholders offered<\/a> to take the company private for $3.46 billion.\u00a0<\/p>\n

JP Morgan Chase reportedly gave<\/a> employees the option to use a chatbot to write performance reviews.\u00a0<\/p>\n

A look at the rise<\/a> of AI-generated receipts and expense fraud. Seniors are turning to<\/a> AI chatbots for companionship. <\/p>\n

\"\"Those good posts<\/p>\n

For more good posts every day, follow Casey\u2019s Instagram stories<\/a>.<\/p>\n

\"\"<\/p>\n

(Link<\/a>)<\/p>\n

\"\"<\/p>\n

(Link<\/a>)<\/p>\n

\"\"<\/p>\n

(Link<\/a>)<\/p>\n

\"\"<\/p>\n

(Link<\/a>)<\/p>\n

\"\"Talk to us<\/p>\n

Send us tips, comments, questions, and mental health stats: casey@platformer.news<\/a>. Read our ethics policy here<\/a>.<\/p>\n