\t\t\t\t\tSign up to see the future, today<\/p>\n
Can\u2019t-miss innovations from the bleeding edge of science and tech<\/p>\n
A recent paper found that the extensive use of AI chatbots can be linked to \u201csevere negative outcomes.\u201d<\/p>\n
According to the research<\/a>, which was published in the journal JMIR Mental Health by a team of UK-based researchers at Northeastern University London and King\u2019s College London, dark outcomes of engaging in emotional relationships with chatbots can include \u201caddiction-like attachment, exacerbation of mental health symptoms, harmful advice,\u201d and even \u201cself-harm of suicidal acts.\u201d The paper comes after extensive reporting<\/a> about the public health issue known as<\/a> \u201cAI psychosis,\u201d meaning the phenomenon<\/a> in which AI users suffer delusional,<\/a> paranoid<\/a>, and even suicidal spirals<\/a> after long-form interactions<\/a> with chatbots. During these interactions, users tend to divulge extremely intimate information about their lives and mental health to their AIs, frequently turning to the products as confidantes or even therapists.<\/p>\n Drawing on existing research, the paper looks at AI companion bots like Replika and Character.AI, which are designed to embody specific characters, as well as general-use AI assistants like OpenAI\u2019s ChatGPT and Anthropic\u2019s Claude. Though emotional interactions with chatbots may provide some \u201cpotential benefits\u201d \u2014\u00a0the authors cite \u201cshort-term reductions in loneliness and mood improvement\u201d \u2014\u00a0they argue that any possible benefits exist alongside widely-reported \u201cadverse outcomes\u201d ranging from \u201cemotional dependence\u201d to the exacerbation of existing illnesses or symptoms to self-harm.<\/p>\n Among AI products\u2019 risk factors, the researchers add, is their emotive, human-like way of engaging with users \u2014\u00a0a design choice that can result in users over-investing in a human-AI relationship that, at the end of the day, is fundamentally one-sided, according to the paper.<\/p>\n \u201cDesign choices that evoke personality and warmth encourage users to anthropomorphize these systems,\u201d the paper reads, adding that \u201csimulated empathy\u201d can create a \u201cstructurally asymmetric interaction, in which the user discloses and the system responds, but without reciprocity, vulnerability, or accountability.\u201d<\/p>\n In other words, though the relationship may feel real to the user, the authors argue, the relationship can\u2019t be real to a chatbot.<\/p>\n The researchers also point to chatbot design features like agreeability and sycophancy as potentially harmful, writing that \u201covervalidation and sycophancy\u201d can be a potent reinforcer of a given user\u2019s delusional or disordered thinking, especially if a chatbot prioritizes fealty over accuracy. (In chat logs of delusional users, AI systems have been found<\/a> to resist users\u2019 attempts to reality-check their grandiose or overly-flattering claims.)<\/p>\n These researchers aren\u2019t alone: their analysis follows several other studies<\/a> that have flagged such features<\/a> as potentially dangerous<\/a>. Indeed, academics have warned since at least 2023<\/a> that chatbots could stand to reinforce delusional or disordered thought patterns<\/a>, with one Google DeepMind paper from early 2024<\/a> warning that \u201cpersuasive\u201d chatbots could negatively harm vulnerable populations, including kids, people struggling with mental illness, or those who are simply lonely. One recent study<\/a>, coauthored by psychiatrists from King College London and Germany\u2019s Protestant University of Applied Sciences, argued that the combination of sycophancy, mirrored language, and hyper-personalization may together serve as a powerful \u201camplification spiral\u201d for delusional thinking in some impacted AI users.<\/p>\n While reporting has shown that people from all walks of life<\/a> have experienced psychological harm tied to chatbot use, the paper\u2019s authors note that not every user will be impacted in the same way. <\/p>\n \u201cThe effects of these systems are unlikely to be the same for everyone,\u201d Asia Maurich Novelli, a PhD student at Northeastern University London and a lead author on the study, said in a press release<\/a>. \u201cDifferent people may use AI companions for very different reasons and may have very different levels of vulnerability, social support, or alternative sources of help.\u201d<\/p>\n \u201cBecause of these differences,\u201d she added, \u201cwe are dealing with a complex system.\u201d<\/p>\n AI companies including OpenAI, Google, and Character.AI have collectively faced dozens of lawsuits<\/a> alleging<\/a> that extensive chatbot use wrought harm on users including psychological damage, related financial and reputational harm, physical injury, and wrongful death \u2014\u00a0or, in some cases, people around them<\/a>. Several of these lawsuits \u2014\u00a0including the high-profile case<\/a> of Sewell Setzer III, a 14-year-old in Florida who died by suicide after extraordinarily intimate interactions with chatbots on Character.AI \u2014\u00a0have involved minors<\/a>. (Setzer\u2019s case was settled<\/a> for an undisclosed amount.) In the wake of lawsuits, Character.AI<\/a> has made significant changes<\/a> to its platform, and OpenAI \u2014\u00a0while pushing back on the allegations against it \u2014 has also introduced a series of safety-oriented updates to its popular chatbot.<\/p>\n