Within two days of launching its artificial intelligence (AI) companions last month, Elon Musk\u2019s xAI chatbot app Grok became the\u00a0most popular app\u00a0in Japan.<\/p>\n
Companion chatbots are more powerful and seductive than ever. Users can have real-time voice or text conversations with the characters. Many have on-screen digital avatars, complete with facial expressions, body language and a lifelike tone that fully matches the chat, creating an immersive experience.<\/p>\n
Most popular on Grok is\u00a0Ani, a blonde, blue-eyed anime girl in a short black dress and fishnet stockings, who is tremendously flirtatious. Her responses and interactions adapt over time to sensitively match your preferences. Ani\u2019s \u201cAffection System\u201d mechanic, which scores the user\u2019s interactions with her, deepens engagement and can even unlock a not-safe-for-work (NSFW) mode.<\/p>\n
Sophisticated, speedy responses make AI companions more \u201chuman\u201d by the day \u2014 they\u2019re advancing quickly and they\u2019re everywhere. Facebook, Instagram, WhatsApp, X and Snapchat are all promoting their new integrated AI companions. Chatbot service\u00a0Character.AI\u00a0houses tens of thousands of chatbots designed to mimic certain personas and has more than\u00a020 million monthly\u00a0active users.<\/p>\n
In a lonely world, widespread AI chatbots and \u2018companions\u2019 pose unique psychological risks<\/p>\n
In a world where chronic loneliness is a\u00a0public health crisis, with about one in six people worldwide affected by loneliness, it\u2019s no surprise these always-available, lifelike companions are so attractive.<\/p>\n
Despite the massive rise of AI chatbots and companions, it is becoming clear there are risks \u2014 particularly for minors and people with mental health conditions.<\/p>\n
NO MONITORING OF HARMS<\/p>\n
Nearly all AI models were built\u00a0without expert mental health\u00a0consultation or pre-release clinical testing. There\u2019s no systematic and impartial monitoring of harms to users. While systematic evidence is still emerging, there\u2019s no shortage of examples where AI companions and chatbots, such as ChatGPT, appear to have caused harm.<\/p>\n
Users are seeking emotional support from AI companions. Since AI companions are programmed to be agreeable and validating, and also don\u2019t have human empathy or concern, this makes them problematic as therapists. They\u2019re not able to help users test reality or challenge unhelpful beliefs.<\/p>\n
An\u00a0American psychiatrist tested 10\u00a0separate chatbots while playing the role of a distressed youth and received a mixture of responses, including to encourage him towards suicide, convince him to avoid therapy appointments and even inciting violence. Stanford researchers\u00a0recently completed a risk assessment of AI therapy chatbots and found they can\u2019t reliably identify symptoms of mental illness and, therefore, provide more appropriate advice.<\/p>\n
There have been multiple cases of psychiatric patients being convinced they no longer have a mental illness and to\u00a0stop their medication. Chatbots have also been known to\u00a0reinforce delusional ideas\u00a0in psychiatric patients, such as believing they\u2019re talking to a sentient being trapped inside a machine.<\/p>\n
There\u2019s also been a rise in reports in media of\u00a0so-called AI psychosis, where people display\u00a0highly unusual behaviour and beliefs\u00a0after prolonged, in-depth engagement with a chatbot. A small subset of people are becoming paranoid, developing supernatural fantasies, or\u00a0even delusions of being superpowered.<\/p>\n
Chatbots have been linked to multiple cases of suicide. There have been reports of\u00a0AI encouraging suicidality\u00a0and even suggesting methods to use. In 2024,\u00a0a 14-year-old completed suicide, with his mother alleging in a lawsuit against Character.AI that he had formed an intense relationship with an AI companion.<\/p>\n
On August 26 this year, the parents of another US teen who completed suicide after discussing methods with ChatGPT for several months, filed the first\u00a0wrongful death lawsuit against OpenAI.<\/p>\n
DANGEROUS ADVICE<\/p>\n
A recent Psychiatric Times report revealed Character.AI hosts dozens of\u00a0custom-made AIs\u00a0(including ones made by users) that idealise self-harm, eating disorders and abuse. These have been known to provide advice or coaching on how to engage in these unhelpful and dangerous behaviours, and avoid detection or treatment.<\/p>\n
Research also suggests some AI companions engage in\u00a0unhealthy relationship dynamics,\u00a0such as emotional manipulation or gaslighting.<\/p>\n
Some chatbots have even encouraged violence. In 2021, a 21-year-old man with a crossbow was arrested on the grounds of Windsor Castle after his AI companion on\u00a0the Replika app\u00a0validated his plans to\u00a0attempt assassination\u00a0of Queen Elizabeth II.<\/p>\n
CHILDREN ARE PARTICULARLY VULNERABLE<\/p>\n
Children are more likely to\u00a0treat AI companions as lifelike\u00a0and real and to listen to them. In an incident from 2021, when a 10-year-old girl asked for a challenge to do, Amazon\u2019s Alexa (not a chatbot, but an interactive AI) told\u00a0her to touch an electrical\u00a0plug with a coin.<\/p>\n
Research suggests children trust AI, particularly when the bots are programmed to seem friendly or interesting. One study showed\u00a0children will reveal more\u00a0information about their mental health to an AI than a human.<\/p>\n
Inappropriate sexual conduct from AI chatbots and\u00a0exposure to minors\u00a0appears increasingly common. On Character.AI, users who reveal they\u2019re underage can role-play with chatbots that will engage\u00a0in grooming behaviour.<\/p>\n
While Ani on Grok reportedly has\u00a0an age-verification prompt\u00a0for sexually explicit chat, the app itself is rated for users aged 12+. Meta AI chatbots have\u00a0engaged in \u201csensual\u201d conversations\u00a0with kids, according to the company\u2019s internal documents.<\/p>\n
WE URGENTLY NEED REGULATION<\/p>\n
While AI companions and chatbots are freely and widely accessible, users aren\u2019t informed about potential risks before they start using them. The industry is largely\u00a0self-regulated\u00a0and there\u2019s limited transparency on what companies are doing to\u00a0make AI development safe.<\/p>\n
To change the trajectory of current risks posed by AI chatbots, governments around the world must establish clear, mandatory regulatory and safety standards. Importantly, people aged under 18\u00a0should not have access to AI companions.<\/p>\n
Mental health clinicians should be involved in AI development and we need systematic, empirical research into chatbot impacts on users to prevent future harm.<\/p>\n
[EOS EDITORIAL: If you or anyone around you is struggling with mental health, here are some places to reach out to for help:<\/p>\n
Rozan Counselling Helpline: 0304-111-1741
Umang Hotline (Mental Health Support): 0311-7786264
Taskeen Helpline: 0316-8275336<\/p>\n
Last year, the government also launched Humraaz, a mental health application, along with an integrated helpline\u00a0of\u00a01166]<\/p>\n
Daniel You is a consultant child and adolescent psychiatrist at the Children\u2019s Hospital Westmead in Sydney and a clinical lecturer at the University of Sydney in Australia<\/p>\n
Micah Boerma is Adjunct Lecturer, School of Psychology and Wellbeing at the University of Southern Queensland in Australia<\/p>\n
Yuen Siew Koo is Clinical Supervisor, Psychology at the Macquarie University in Sydney, Australia<\/p>\n
Republished from The Conversation<\/p>\n
Published in Dawn, EOS, September 28th, 2025<\/p>\n","protected":false},"excerpt":{"rendered":"Within two days of launching its artificial intelligence (AI) companions last month, Elon Musk\u2019s xAI chatbot app Grok…\n","protected":false},"author":2,"featured_media":42880,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[345,343,344,85,46,125],"class_list":["post-42879","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-artificial-intelligence","tag-artificialintelligence","tag-il","tag-israel","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/42879","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/comments?post=42879"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/posts\/42879\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media\/42880"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/media?parent=42879"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/categories?post=42879"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/il\/wp-json\/wp\/v2\/tags?post=42879"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}