At least once a month, two-thirds of people who regularly use AI turn to their bots for advice on sensitive personal issues and emotional support.<\/p>\n
Many people now report trusting their chatbots more than their elected representatives, civil servants, faith leaders\u2014and the companies building AI. That\u2019s according to data<\/a> from 70 countries, gathered by the Collective Intelligence Project<\/a> (CIP). As CIP\u2019s research director, neuroscientist Zarinah Agnew<\/a>, puts it, AI is becoming \u201cemotional infrastructure at scale.\u201d And it\u2019s being built by companies whose economic incentives may not align with our wellbeing.<\/p>\n Already, we\u2019ve seen instances of AI companies optimizing their models to keep people engaged, even when this goes against their best interests. Last April, OpenAI had to roll back<\/a> an update<\/a> to one of its ChatGPT models after it was widely-criticized for being overly-flattering to users. When the company stopped offering the model to people, the day before Valentine\u2019s Day, some were distraught<\/a>.<\/p>\n Humans finding comfort in machines is not new. In the late 1990s, MIT Professor Rosalind Picard<\/a>\u2014who founded the field of affective computing\u2014found that people responded positively to computers performing empathy. But two key things have changed since then: thanks to technical advances, AI systems today are new entities<\/a>, capable of sophisticated conversation and surprising behavior; and thanks to the billions of dollars investors have poured into AI companies, these entities are accessible to virtually anyone with an internet connection. ChatGPT alone currently has<\/a> more than 800 million weekly active users\u2014and the number is growing.<\/p>\n But with millions of people forming different kinds of human-machine relationships, we don\u2019t yet know whether AI is helping more people than it harms. And meanwhile, AI companies are investing in making their models not just smarter, but also more emotionally savvy\u2014better at detecting emotion in a person\u2019s voice, and at responding appropriately. People are trusting their chatbots with deeply personal information, even while they distrust the companies creating them, and while the companies are exploring advertising and other revenue models to sustain themselves.<\/p>\n \u201cI think we may have a crisis on our hands,\u201d says Picard.<\/p>\n Emotional beings<\/p>\n Humans are inherently social. “We don\u2019t do well\u2014biologically, immunologically, neurally, or politically\u2014when we\u2019re in isolation,\u201d says Agnew. Today\u2019s AI systems have arrived at a time when \u201cwe\u2019ve largely failed to provision for intimacy for most people\u2014both in terms of what the state can provide and human sociality,” they say.\u00a0<\/p>\n AI is effective at providing emotional support because it offers an approximation of what Professor Marc Brackett<\/a>\u2014head of the Yale Center for Emotional Intelligence<\/a>\u2014calls \u201cpermission to feel,\u201d which he argues is foundational in learning to process emotions. Adults who provide this permission are \u201cnon-judgmental people who are good listeners and show empathy and compassion.\u201d In 70 studies Brackett has conducted across the world, only around 35% of people report having had an adult like that around when they were kids. Chatbots, which are non-judgmental, compassionate, and always available, can provide permission to feel at scale.<\/p>\n Lisa Feldman-Barrett<\/a>, a psychology professor at Northeastern University, says \u201csocial support from a trusted, reliable source can be beneficial.\u201d If an AI can reduce distress in the moment, she says that\u2019s a good thing. But healthy human relationships\u2014platonic or therapeutic\u2014do more than comfort. They challenge. A good therapist helping you change your behavior, she says, will \u201chold your feet to the fire.\u201d<\/p>\n But AI models vary in how much they meaningfully challenge their users\u2014particularly since different models perform different personalities, each of which changes slightly with each new release. The ChatGPT sycophancy episode showed that some users may prefer models that flatter them over ones that offer a challenge. So companies looking to maximize engagement with their chatbots may prefer to tweak them to pander.\u00a0<\/p>\n Whether AI models themselves are truly emotionally intelligent is academically contested\u2014as is the definition of emotion itself. But as Picard points out, while the question of what defines emotions, and whether AI can truly be said to have them, now or in future, is interesting, \u201cwe don\u2019t need [to answer] it to build systems that have emotional intelligence.\u201d<\/p>\n The AI companies, of course, already know this. \u201cThe extent of anthropomorphism in any given AI is just a design decision to be taken by the AI\u2019s developer\u2014who faces many commercial incentives to increase it,\u201d Google DeepMind researchers wrote in an October 2025 paper<\/a>. The same paper notes that \u201cthe emotional vulnerabilities tied to loneliness can make individuals more susceptible to manipulation by AIs engineered to foster dependence and one-sided attachment,\u201d and that \u201cthe absence of rigorous, long-term studies on the effects of AI companionship means we are still largely in the dark concerning the potential for adverse outcomes.\u201d\u00a0<\/p>\n Mixed signals<\/p>\n Voice is the next frontier. As AI systems become better at recognizing human emotion, and speaking expressively, our relationships with them may deepen. Under increasing pressure to generate revenue, AI companies may lean into developing their models in ways that foster emotional dependence. After OpenAI announced it would begin testing ads in ChatGPT, former OpenAI researcher Zo\u00eb Hitzig resigned, writing<\/a> in The New York Times that she was concerned the company\u2014like social media companies before it\u2014may veer from its self-imposed commitments<\/a> around advertising. \u201cThe company is building an economic engine that creates strong incentives to override its own rules,\u201d she wrote.\u00a0<\/p>\n