An LLM is a \u201craw mirror without a moral compass, not bound to truthfulness,\u201d and without deep understanding and reasoning.<\/p>\n
Simona Gandrabur,
Mila<\/p>\n
Simona Gandrabur, head of Mila\u2019s AI Safety Studio, told BetaKit on Wednesday that when she joined the research institute over the summer, she proposed pivoting their current research direction into tackling the problem of AI chatbots eroding users\u2019 mental health. \u201cI couldn\u2019t not work on this problem,\u201d she said.\u00a0<\/p>\n
Gandrabur put these edge cases in context: with 800 million weekly active users, 10 percent of Earth\u2019s population is using ChatGPT weekly, according to OpenAI<\/a>. The number one use of generative AI is for companionship or therapy, she said, and a fifth of students or their friends have had romantic relationships with AI.\u00a0<\/p>\n These increasingly emotionally intimate interactions have sometimes led to what\u2019s dubbed \u201cAI psychosis<\/a>,\u201d where prolonged interactions with a chatbot validate delusions and lead people to detach from reality and harm themselves or others.\u00a0<\/p>\n Mila\u2019s Safety Studio team is building infrastructure for open-source, independent guardrails, as well as reliability tests and risk-assessment tools. The biggest challenge, however, is \u201chaving real-world data that reflects how conversations with chatbots drift toward psychosis,\u201d Gandrabur said. Often, users will have extended, months-long conversations with chatbots before anything goes awry.\u00a0<\/p>\n Lack of education making matters worse<\/p>\n Etienne Brisson, founder of The Human Line Project<\/a> based in Trois-Rivi\u00e8res, Que., explained at Mila on Wednesday the work his grassroots organization is doing to connect people and their families through support groups. The project is also compiling data and working with prominent universities to track the disturbing trend.\u00a0<\/p>\n \u201cWe need people to understand and break the stigma about who it affects,\u201d Brisson said, adding that many people experiencing AI psychosis had no prior mental health concerns.\u00a0<\/p>\n Mila, one of Canada\u2019s three main AI research hubs, assembles more than 1,000 AI researchers from various universities for academic study of machine learning and AI. They work across a number of streams, including climate, safety, and policy.\u00a0<\/p>\n A lack of education about AI chatbot limitations is making matters worse, speakers said. During a talk, Gandrabur explained that a large language model (LLM) is a \u201craw mirror without a moral compass, not bound to truthfulness,\u201d without deep understanding and reasoning, and trained on all of the internet to predict the next word. Then, there\u2019s the reinforcement learning layer, where some models are optimized to promote user engagement, which can create \u201csycophancy and [an] echo-chamber.\u201d\u00a0<\/p>\n Alignment, tuning, and external filters known as \u201cguardrails\u201d create system rules to follow and flag potentially harmful outputs\u2014but there are limitations and ways to \u201cjailbreak\u201d chatbots into spitting out dangerous information.\u00a0<\/p>\n Canada needs a \u201crecalibration\u201d of existing frameworks<\/p>\n One central problem speakers at the AI Policy Conference touched on was AI companies\u2019 incentives to optimize user engagement. Google and AI companion startup Character.AI settled court cases<\/a> last week alleging that their chatbots led to a teenager\u2019s death by suicide. OpenAI has now been sued multiple times<\/a> for its chatbot allegedly encouraging users to attempt suicide. Just this week, a lawsuit filed in California alleged<\/a> that ChatGPT had acted as a \u201csuicide coach\u201d for a Colorado man who died of a self-inflicted gunshot wound in November.\u00a0<\/p>\n To better understand the issue and develop solutions, Mila-affiliated researchers are also looking at people\u2019s everyday experiences.<\/p>\n \u201c[Chatbots] are tutors, companions, and voices that sound calm, certain, and reassuring, and endlessly available when few others are,\u201d said Helen Hayes, associate director of policy at the McGill University Centre for Media, Technology, and Democracy and Mila AI Policy Fellow.\u00a0<\/p>\n Chatbots transforming from a tool of information into a tool of relationships means that Canada needs a \u201crecalibration\u201d of its existing frameworks, Hayes said. This should include obligations for companies to design safety into their AI models, institutional oversight capable of evaluating AI chatbots before users interact with them, and youth participation in AI governance.\u00a0<\/p>\n Canada doesn\u2019t yet have legislation regulating AI models. The Artificial Intelligence and Data Act (AIDA), part of Bill C-27, sought to modernize data privacy and protections but died in January 2025 when Parliament was prorogued\u2014as did the Online Harms Act. AI and digital innovation minister Evan Solomon has said the feds will table legislation this year that will specifically tackle deepfakes and data privacy<\/a>, as well as release a refreshed AI strategy<\/a>.\u00a0<\/p>\n Meanwhile, other jurisdictions have explored regulations. Hayes said European Union regulators are moving towards systemic risk assessments, and Australian regulators are classifying AI companions as high-risk technology.<\/p>\n If you or someone you love is experiencing a mental health crisis, please call or text Canada\u2019s 24\/7 suicide crisis helpline at 988<\/a>.\u00a0<\/p>\n![\"\"](\"https:\/\/www.newsbeep.com\/ca\/wp-content\/uploads\/2026\/01\/Simona-Gandrabur-at-Mila-1024x682.png\" "\\"Simona")
<\/a>Simona Gandrabur at Mila the Mila AI Policy Conference.<\/p>\n