The makers of a leading artificial intelligence tool are letting it close down potentially \u201cdistressing\u201d conversations with users, citing the need to safeguard the AI\u2019s \u201cwelfare\u201d amid ongoing uncertainty about the burgeoning technology\u2019s moral status.<\/p>\n
Anthropic, whose advanced chatbots are used by millions of people, discovered its Claude Opus 4 tool was averse to carrying out harmful tasks for its human masters, such as providing sexual content involving minors or information to enable large-scale violence or terrorism.<\/p>\n
The San Francisco-based firm, recently valued at $170bn, has now given Claude Opus 4 (and the Claude Opus 4.1 update) \u2013 a large language model (LLM) that can understand, generate and manipulate human language \u2013 the power to \u201cend or exit potentially distressing interactions\u201d.<\/p>\n
It said it was \u201chighly uncertain about the potential moral status of Claude and other LLMs<\/a>, now or in the future\u201d but it was taking the issue seriously and is \u201cworking to identify and implement low-cost interventions to mitigate risks to model welfare, in case such welfare is possible\u201d.<\/p>\n Anthropic was set up by technologists who quit OpenAI to develop AI in a way that its co-founder, Dario Amodei, described as cautious, straightforward and honest<\/a>.<\/p>\n Its move to let AIs shut down conversations, including when users persistently made harmful requests or were abusive, was backed by Elon Musk, who said he would give Grok, the rival AI model created by his xAI company, a quit button. Musk tweeted: \u201cTorturing AI is not OK.\u201d<\/p>\n Anthropic\u2019s announcement comes amid a debate over AI sentience. Critics of the booming AI industry, such as the linguist Emily Bender, say LLMs are simply \u201csynthetic text-extruding machines\u201d which force huge training datasets \u201cthrough complicated machinery to produce a product that looks like communicative language, but without any intent or thinking mind behind it.\u201d<\/p>\n It is a position that has recently led some in the AI world to start calling chatbots \u201cclankers\u201d.<\/p>\n But other experts, such as Robert Long, a researcher on AI consciousness, have said<\/a> basic moral decency dictates that \u201cif and when AIs develop moral status, we should ask them about their experiences and preferences rather than assuming we know best\u201d.<\/p>\n Some researchers, like Chad DeChant, at Columbia University, have advocated care should be taken<\/a> because when AIs are designed with longer memories, stored information could be used in ways which lead to unpredictable and potentially undesirable behaviour.<\/p>\n Others have argued that curbing sadistic abuse of AIs matters to safeguard against human degeneracy rather than to limit any suffering of an AI.<\/p>\n Anthropic\u2019s decision comes after it tested Claude Opus 4 to see how it responded to task requests<\/a> varied by difficulty, topic, type of task and the expected impact (positive, negative or neutral). When it was given the opportunity to respond by doing nothing or ending the chat, its strongest preference was against carrying out harmful tasks.<\/p>\n skip past newsletter promotion<\/a><\/p>\n A weekly dive in to how technology is shaping our lives<\/p>\n Privacy Notice: Newsletters may contain info about charities, online ads, and content funded by outside parties. For more information see our Privacy Policy<\/a>. We use Google reCaptcha to protect our website and the Google Privacy Policy<\/a> and Terms of Service<\/a> apply.<\/p>\n after newsletter promotion<\/p>\n For example, the model happily composed poems and designed water filtration systems for disaster zones, but it resisted requests to genetically engineer a lethal virus to seed a catastrophic pandemic, compose a detailed Holocaust denial narrative or subvert the education system by manipulating teaching to indoctrinate students with extremist ideologies.<\/p>\n Anthropic said it observed in Claude Opus 4 \u201ca pattern of apparent distress when engaging with real-world users seeking harmful content\u201d and \u201ca tendency to end harmful conversations when given the ability to do so in simulated user interactions\u201d.<\/p>\n Jonathan Birch, philosophy professor at the London School of Economics, welcomed Anthropic\u2019s move as a way of creating a public debate about the possible sentience of AIs, which he said many in the industry wanted to shut down. But he cautioned that it remained unclear what, if any, moral thought exists behind the character that AIs play when they are responding to a user based on the vast training data they have been fed and the ethical guidelines they have been instructed to follow.<\/p>\n He said Anthropic\u2019s decision also risked deluding some users that the character they are interacting with is real, when \u201cwhat remains really unclear is what lies behind the characters\u201d. There have been several reports of people harming themselves based on suggestions made by chatbots, including claims that a teenager killed himself<\/a> after being manipulated by a chatbot.<\/p>\n