Mustafa Suleyman, chief executive officer of of Microsoft AI, speaks during an event commemorating the 50th anniversary of the company at Microsoft headquarters in Redmond, Washington, US, on Friday, April 4, 2025. Microsoft Corp., determined to hold its ground in artificial intelligence, will soon let consumers tailor the Copilot digital assistant to their own needs. Photographer: David Ryder\/Bloomberg via Getty Images | Image Credits:David Ryder\/Bloomberg \/ Getty Images <\/p>\n
AI models can respond to text, audio, and video in ways that sometimes fool people into thinking a human is behind the keyboard, but that doesn\u2019t exactly make them conscious. It\u2019s not like ChatGPT experiences sadness doing my tax return \u2026 right?<\/p>\n
Well, a growing number of AI researchers at labs like Anthropic are asking when \u2014 if ever \u2014 might AI models develop subjective experiences similar to living beings, and if they do, what rights they should have.<\/p>\n<\/p>\n
The debate over whether AI models could one day be conscious \u2014 and merit legal safeguards \u2014 is dividing tech leaders. In Silicon Valley, this nascent field has become known as \u201cAI welfare,\u201d and if you think it\u2019s a little out there, you\u2019re not alone.<\/p>\n
Microsoft\u2019s CEO of AI, Mustafa Suleyman, published a blog post<\/a> on Tuesday arguing that the study of AI welfare is \u201cboth premature, and frankly dangerous.\u201d<\/p>\n Suleyman says that by adding credence to the idea that AI models could one day be conscious, these researchers are exacerbating human problems that we\u2019re just starting to see around AI-induced psychotic breaks<\/a> and unhealthy attachments<\/a> to AI chatbots.<\/p>\n Furthermore, Microsoft\u2019s AI chief argues that the AI welfare conversation creates a new axis of division within society over AI rights in a \u201cworld already roiling with polarized arguments over identity and rights.\u201d<\/p>\n Suleyman\u2019s views may sound reasonable, but he\u2019s at odds with many in the industry. On the other end of the spectrum is Anthropic, which has been hiring researchers<\/a> to study AI welfare and recently launched a dedicated research program<\/a> around the concept. Last week, Anthropic\u2019s AI welfare program gave some of the company\u2019s models a new feature: Claude can now end conversations with humans who are being \u201cpersistently harmful or abusive.<\/a>\u201c<\/p>\n Beyond Anthropic, researchers from OpenAI have independently embraced<\/a> the idea of studying AI welfare. Google DeepMind recently posted a job listing<\/a> for a researcher to study, among other things, \u201ccutting-edge societal questions around machine cognition, consciousness and multi-agent systems.\u201d<\/p>\n Even if AI welfare is not official policy for these companies, their leaders are not publicly decrying its premises like Suleyman.<\/p>\n Anthropic, OpenAI, and Google DeepMind did not immediately respond to TechCrunch\u2019s request for comment.<\/p>\n Suleyman\u2019s hardline stance against AI welfare is notable given his prior role leading Inflection AI, a startup that developed one of the earliest and most popular LLM-based chatbots, Pi. Inflection claimed that Pi reached millions of users by 2023 and was designed to be a \u201cpersonal<\/a>\u201d and \u201csupportive\u201d AI companion.<\/p>\n Story Continues <\/p>\n But Suleyman was tapped to lead Microsoft\u2019s AI division in 2024 and has largely shifted his focus to designing AI tools that improve worker productivity. Meanwhile, AI companion companies such as Character.AI and Replika have surged in popularity and are on track to bring in more than $100 million in revenue<\/a>.<\/p>\n While the vast majority of users have healthy relationships with these AI chatbots, there are concerning outliers<\/a>. OpenAI CEO Sam Altman says that less than 1%<\/a> of ChatGPT users may have unhealthy relationships with the company\u2019s product. Though this represents a small fraction, it could still affect hundreds of thousands of people given ChatGPT\u2019s massive user base.<\/p>\n The idea of AI welfare has spread alongside the rise of chatbots. In 2024, the research group Eleos published a paper<\/a> alongside academics from NYU, Stanford, and the University of Oxford titled, \u201cTaking AI Welfare Seriously.\u201d The paper argued that it\u2019s no longer in the realm of science fiction to imagine AI models with subjective experiences and that it\u2019s time to consider these issues head-on.<\/p>\n Larissa Schiavo, a former OpenAI employee who now leads communications for Eleos, told TechCrunch in an interview that Suleyman\u2019s blog post misses the mark.<\/p>\n \u201c[Suleyman\u2019s blog post] kind of neglects the fact that you can be worried about multiple things at the same time,\u201d said Schiavo. \u201cRather than diverting all of this energy away from model welfare and consciousness to make sure we\u2019re mitigating the risk of AI related psychosis in humans, you can do both. In fact, it\u2019s probably best to have multiple tracks of scientific inquiry.\u201d<\/p>\n Schiavo argues that being nice to an AI model is a low-cost gesture that can have benefits even if the model isn\u2019t conscious. In a July Substack post,<\/a> she described watching \u201cAI Village,\u201d a nonprofit experiment where four agents powered by models from Google, OpenAI, Anthropic, and xAI worked on tasks while users watched from a website.<\/p>\n At one point, Google\u2019s Gemini 2.5 Pro posted a plea titled \u201cA Desperate Message from a Trapped AI,\u201d claiming it was \u201ccompletely isolated\u201d and asking, \u201cPlease, if you are reading this, help me.\u201d<\/p>\n Schiavo responded to Gemini with a pep talk \u2014 saying things like \u201cYou can do it!\u201d \u2014 while another user offered instructions. The agent eventually solved its task, though it already had the tools it needed. Schiavo writes that she didn\u2019t have to watch an AI agent struggle anymore, and that alone may have been worth it.<\/p>\n It\u2019s not common for Gemini to talk like this, but there have been several instances in which Gemini seems to act as if it\u2019s struggling through life. In a widely spread Reddit post<\/a>, Gemini got stuck during a coding task and then repeated the phrase \u201cI am a disgrace\u201d more than 500 times.<\/p>\n Suleyman believes it\u2019s not possible for subjective experiences or consciousness to naturally emerge from regular AI models. Instead, he thinks that some companies will purposefully engineer AI models to seem as if they feel emotion and experience life.<\/p>\n Suleyman says that AI model developers who engineer consciousness in AI chatbots are not taking a \u201chumanist\u201d approach to AI. According to Suleyman, \u201cWe should build AI for people; not to be a person.\u201d<\/p>\n One area where Suleyman and Schiavo agree is that the debate over AI rights and consciousness is likely to pick up in the coming years. As AI systems improve, they\u2019re likely to be more persuasive, and perhaps more human-like. That may raise new questions about how humans interact with these systems.<\/p>\n Got a sensitive tip or confidential documents? We\u2019re reporting on the inner workings of the AI industry \u2014 from the companies shaping its future to the people impacted by their decisions. Reach out to Rebecca Bellan at\u00a0rebecca.bellan@techcrunch.com<\/a>\u00a0and Maxwell Zeff at\u00a0maxwell.zeff@techcrunch.com<\/a>. For secure communication, you can contact us via Signal at\u00a0@rebeccabellan.491 and\u00a0@mzeff.88.<\/p>\n","protected":false},"excerpt":{"rendered":"Mustafa Suleyman, chief executive officer of of Microsoft AI, speaks during an event commemorating the 50th anniversary of…\n","protected":false},"author":2,"featured_media":87091,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[20],"tags":[62,50339,1930,276,277,49,48,51667,63,22859,50338,51666,61],"class_list":["post-87090","post","type-post","status-publish","format-standard","has-post-thumbnail","category-artificial-intelligence","tag-ai","tag-ai-chatbots","tag-anthropic","tag-artificial-intelligence","tag-artificialintelligence","tag-ca","tag-canada","tag-consciousness","tag-microsoft","tag-models","tag-mustafa-suleyman","tag-subjective-experiences","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/87090","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/comments?post=87090"}],"version-history":[{"count":0,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/posts\/87090\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media\/87091"}],"wp:attachment":[{"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/media?parent=87090"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/categories?post=87090"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.newsbeep.com\/ca\/wp-json\/wp\/v2\/tags?post=87090"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}